Simulating Training Dynamics to Reconstruct Training Data from Deep Neural Networks

Thank you for your appreciation of our performance and clarity, along with your insightful suggestions. We address your concerns as below:

R4.1 Setting Definition.

Thank you for your careful reading and valuable suggestions. We will elaborate on the setting for dataset reconstruction.

We have access to only the initial and final parameters of the model. And, we manage to reconstruct training dataset from the two sets of parameters.

We noticed that you might have a slight misunderstanding regarding our method: you mentioned that we require full access to hyper-parameters. Having access to the training hyper-parameters will help reconstruction, but for more common situations where hyper-parameters are unknown, we have designed a strategy to find suitable hyper-parameters for simulation of dynamics. Specifically, we preset the batch size and tune learning rate by grid search based on the first dozens of steps’ loss. Thus, suitable hyper-parameters can be identified effectively and efficiently. The strategy is introduced in Section 3, and the reconstruction results of different hyper-parameters identified using our strategy are shown in Section 4.3 Fig.7.

The two sets of parameters of the model are accessible in many realistic scenarios, such as fine-tuning from public models, or in federated learning where clients receive period updates of the model parameters, so we choose such setting. In those scenarios, parameters of model are published alongside model architecture, and only publishing parameters is meaningless without the architecture. In short, our setting is to reconstruct training dataset from only the initial and final parameters.

R4.2 Model Depth, Model-to-Data Ratio & Dataset Size.

• Challenges of Model Depth

The model depth and dataset size both pose great challenges to dataset reconstruction from the parameters. We first discuss the challenges of model depth. Previous works mainly focus on simple and shallow MLPs, which possess linear training dynamics. However, for deeper and more complex DNNs, the training dynamics become non-linear. We propose to simulate such non-linear dynamics for more challenging and practical reconstruction, which constitutes our main contribution.

• Model-to-Data Ratio

Your question about the overfitted model-to-data ratio is quite insightful. All deep models have sufficient parameters, and a very high model-to-data ratio (in other words, over-parameterization) is necessary for benign overfitting [1, 2]: a model fits the training data well while also generalizing effectively to test data. Memorization also needs a sufficient number of parameters. Therefore, we believe that a high model-to-data ratio is the nature of deep learning and meanwhile is necessary for training data reconstruction.

• Challenges of Dataset Size

Dataset size makes reconstruction challenging primarily from two aspects. Firstly, the larger dataset means more GPU memory usage in simulating the training dynamcis and back-propagating with multiple training steps. Secondly, as the dataset size increases, the uncertainty in the optimization problem of decoupling gradients of more data increases, making reconstruction much more challenging. We mention a similar and easier task called gradient inversion attack, where attackers reconstruct training data from one single gradient. Even without considering decoupling gradients from parameters across different steps, the state-of-the-art methods [3, 4] can reconstruct only about 100 images. The decoupling problem is quite hard to optimize. And we can achieve comparable number of reconstruction images even in the more difficult task.

Thank you for your suggestions to reconstruct a subset of training dataset. The idea is very valuable and meaningful. However, as we mentioned above, the more intensive memory requirements and harder optimization of the decoupling problem make it much more challenging. We attempted to optimize only dozens of dummy images to represent a meaningful subset of full dataset. However, the dummy images became a fusion of multiple original images, similar to the phenomenon observed in Section 4.1, making the reconstructed images appear meaningless. During the rebuttal period, we couldn't come up with a better method to reconstruct a subset of the full dataset. Your idea is very meaningful for deep learning and it deserves further investigation in the future. We also believe that more effective solutions of the optimization problem and efficient simulation for less memory requirements will help solve larger datasets in the future.

Thank you for your appreciation of performance and novelty of SimuDy. We address your concerns as below:

R3.1 Practicality & Scalability.

The task of dataset reconstruction from parameters of DNNs is quite challenging. Compared to previous works [1, 2, 3] in dataset reconstruction from parameters, we extend dataset reconstruction from shallow and small MLPs to more pratical DNNs. Moreover, our method SimuDy can generalize well to larger models and higher-resolution datasets, with qualitative recosntruction results of a Vision Transformer trained on ImageNet shown in Appendix C.5 Fig.19.

We propose to simulate non-linear training dynamics, and as you mentioned, back propagating with multiple training steps is computationally intensive and thus makes scale for larger-size datasets challenging. For a similar but simpler task called the gradient inversion attack (GIA), where attackers reconstruct images from a single gradient, state-of-the-art methods [4, 5] can reconstruct approximately 100 images. We achieve a comparable number of reconstructed images, even for this more challenging task. Moreover, our primary motivation is to verify the memorization of DNNs, rather than to attack DNNs in practice. We believe that our methods would provide valuable insights into the memorization of DNNs.

We noticed that there might be a slight misunderstanding regrading our proposed method. Provided only the initial and final parameters, SimuDy can reconstruct training data. Prior knowledge of hyper-parameters, such as the number of training steps, batch size and learning rate, is not necessary for reconstruction. We have developed a strategy to find suitable hyper-parameters for simulating training dynamics, with the details in Section 3 and experimental results in Section 4.3.

[1] Haim N, Vardi G, Yehudai G, et al. Reconstructing training data from trained neural networks. NeurIPS, 2022.

[2] Buzaglo G, Haim N, Yehudai G, et al. Deconstructing data reconstruction: Multiclass, weight decay and general losses. NeurIPS, 2024.

[3] Loo N, Hasani R, Lechner M, et al. Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation. ICLR, 2024.

[4] Yin H, Mallya A, Vahdat A, et al. See through gradients: Image batch recovery via gradinversion. CVPR, 2021.

[5] Zhang C, Xiaoman Z, Sotthiwat E, et al. Generative gradient inversion via over-parameterized networks in federated learning. CVPR, 2023.

R3.2 Implication & Usefulness.

We analyze the linearity of training dynamics for MLPs and ResNets, as shown in Appendix A. We also design a metric to measure the linearity. We consider simulating non-linear training dynamics as a tool for reconstruction from parameters of DNNs. The success of SimuDy indicates that the parameters and the data are intrinsically linked through the training dynamics. Moreover, a better understanding of training dynamics will aid reconstruction, and it would be interesting to explore these dynamics further, such as leveraging the low-dimensional nature of the training process [1, 2] to enable efficient simulation.

Deep learning has achieved marvelous success. However, the mechanism, especially why over-parameterized DNNs can generalize quite well to unseen data, is still unclear. Memorization is closely related to generalization [3] and raises concerns about the potential leakage of private information within training data.

Therefore, verifying or falsifying the presence of memorization in deep models is crucial. Recently, there have been many discussions on membership inference attacks (MIA) [4], which aim to determine whether a specific data point was included in the training dataset. MIA serves as an indicator of whether the model has memorized the data point.

More direct evidence is to reconstruct the training set from a model's parameters. Therefore, pioneering works in this field have attracted significant attention. Our contribution is that we have overcome the limitations of the linearity of the training process. This step is quite significant since real-world training is certainly non-linear. SimuDy can then be used to extract memorization from ResNet, as well as ViT and tinyBERT. We think SimuDy can indeed confirm that DNNs can remember some training samples, which may lead to further exploration into the relationship between memorization and generalization, interpretability, as well as security and data privacy.

[1] Gur-Ari G, Roberts D A, Dyer E. Gradient descent happens in a tiny subspace. arXiv, 2018.

[2] Li T, Tan L, Huang Z, et al. Low dimensional trajectory hypothesis is true: Dnns can be trained in tiny subspaces. TPAMI, 2022.

[3] Feldman V, Zhang C. What neural networks memorize and why: Discovering the long tail via influence estimation. NeurIPS, 2021.

[4] Shokri R, Stronati M, Song C, et al. Membership inference attacks against machine learning models. IEEE S&P, 2017.

Thank you for your appreciation of our performance and clarity, along with your insightful suggestions. We address your concerns as below:

R2.1 Known Initilization Parameters.

Thank you for recognizing that access to initial parameters is not a critical issue. Our main motivation and contribution is not to attack DNNs in practice but to verify the memorization of DNNs. SimuDy is more like an alarm for data security, rather than a practical threat.

We agree with you that knowing the initial parameters is unrealistic in some scenarios. However, access to initial parameters arises naturally in many settings, such as fine-tuning from public models or in federated learning where clients receive period updates of the model parameters.

R2.2 Computational and Memory Budget.

Thank you for your understanding. Following your suggestion, we have reported the reconstruction time and GPU memory usage of SimuDy on CIFAR-10 for different-sized datasets in Appendix C.7 Tab.3, also shown following. Notice that for reconstruction of 120 images, we almost run out of the GPU memory on an RTX 4090 (24GB).

R2.3.1 Comparison with Pioneering Works.

Thank you for your valuable suggestions. Haim et al. [1] first show that the parameters of trained networks contain information to reconstruct training samples. They successfully reconstruct a dataset from a binary MLP. Buzaglo et al. [2] extend reconstruction to a multi-class setting. Both of these pioneering works inspire us greatly. We take a step toward reconstructing training data from the parameters of more complex DNNs. The essential progress is that we can reconstruct data from non-linear training dynamics, whereas previous works are effective only for linear dynamics.

The qualitative and quantitative reconstruction results of Buzaglo et al.'s method have been updated in Appendix C.4 since our method mainly focuses on the multi-class setting. When applied to ResNet-18 with non-linear training dynamics, Buzaglo et al. use only the gradients of the final models to characterize the entire dynamics, which is inadequate and results in degraded reconstruction quality. Moreover, their method requires very small initial parameters or the application of weight decay during training, and the absence of these factors could further degrade performance.

[1] Haim N, Vardi G, Yehudai G, et al. Reconstructing training data from trained neural networks. NeurIPS, 2022.

[2] Buzaglo G, Haim N, Yehudai G, et al. Deconstructing data reconstruction: Multiclass, weight decay and general losses. NeurIPS, 2024.

R2.3.2 Generality of SimuDy on Different Datasets and Model architectures.

Thank you for your valuable suggestions on verifying the generality of SimuDy. We had previously included experiments on another dataset, Tiny ImageNet, and the results can be found in Appendix C.1, Fig. 12. To further verify the generality of SimuDy, we conducted new experiments:

• SVHN dataset and ResNet-50 architecture: The reconstruction results have been reported in Appendix C.2, Fig.13 and Fig.14.

• ViT trained on ImageNet : The reconstruction results have been reported in Appendix C.5, Fig.19.

• TinyBERT trained on CoLA in NLP: The reconstruction results have been reported Appendix C.6, Tab.2.

All the above experiments show that SimuDy can generalize well to a variety of datasets and model architectures.

R2.4 The Affect of Coefficient of TV Loss.

To answer your question, we discretely selected the coefficient $\alpha$ of the TV loss. Both the qualitative and quantitative reconstruction results can be seen in Appendix C.3.

As the coefficient $\alpha$ increases, the quantitative SSIM value initially improves, reaching a peak, and then decreases thereafter. Thus, selecting an appropriate coefficient is crucial, as both excessively large and small values negatively affect reconstruction quality. Interestingly, an excessively large coefficient can cause reconstructed images to become blurry, even resulting in color shifts compared to the original training images, as shown in Appendix C.3 Fig. 17.

Thank you for your appreciation of the novelty of SimuDy and insightful suggestions. We address your concerns as below:

R1.1 Privacy & Security.

We sincerely appreciate your recognition of SimuDy's performance. Reconstruction from parameters is a direct evidence that DNNs remember something through training. This could reveal the mechanism of deep learning and meanwhile pose challenges for privacy and data security, as you concerned.

Pioneering works rely on linear training dynamics. We think stepping over the gap between linear and non-linear dynamics is quite essential towards the real-world DNNs, which is the motivation of our work. Frankly speaking, we did not anticipate that the reconstruction results would be so impressive, both quantitatively and qualitatively, even without precise knowledge of the training hyper-parameters.

Indeed, towards practical implications, there are still many aspects that warrant further consideration and resolution, including the number of training samples, the significant computational costs, and the substantial memory requirements. We hope SimuDy's successful reconstruction could open a door towards the memorization of real-world DNNs and serve as an alarm on data security, rather than an immediate threat.

R1.2 Computational Costs & Memory Requirements.

Indeed, the computational costs and memory requirements of SimuDy are currently heavy. This is because dataset reconstruction from DNNs with non-linear training dynamics is quite difficult and requires storing the entire training computation graph, as you mentioned.

We ourselves regard SimuDy as the first step to extract memory from real-world DNNs, rather than focusing on immediate attack performance.

And we do agree with the reviewer that it will be interesting and meaningful to explore methods of reducing computational resource demands to make SimuDy more practical. We believe that SimuDy will benefit from that, especially considering that attackers are not likely to be frugal with computational resources.

R1.3 Initialization of Dummy Datasets.

According to your valuable suggestion, besides random noise and natural images, we add two initializations that have more priors: one is the original training images with added noise, and the other is original images where the central 16x16 pixels are set to white. The order of reconstruction quality is as follows: original images with added noise > noise > original images with central 16x16 pixels white > other natural images. The convergence speed follows the same order.

The initializations with priors should help better reconstruction and speed up convergence. But original images with central 16x16 pixels white resulted in worse performance than using noise as the initialization. This indicates that the similarity in the view of humans and DNNs is not the same, which makes prior design quite hard.

Prior knowledge of the original dataset is typically unavailable, making it reasonable to use noise as initialization. Most data recovery methods [1, 2, 3, 4] based on optimization use random noise as initialization. SimuDy can successfully reconstruct training data from different initializations.

[1] Zhu L, Liu Z, Han S. Deep leakage from gradients. NeurIPS, 2019.

[2] Geiping J, Bauermeister H, Dröge H, et al. Inverting gradients-how easy is it to break privacy in federated learning? NeurIPS, 2020.

[3] Haim N, Vardi G, Yehudai G, et al. Reconstructing training data from trained neural networks. NeurIPS, 2022.

[4] Loo N, Hasani R, Lechner M, et al. Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation. ICLR, 2024.