Reconstructing Training Data From Real-World Models Trained with Transfer Learning

We thank the reviewer for the thorough feedback.

Answers for “Weaknesses” part:

• This is a good point. Previous works were successful in reconstructing training data only from very small-scale networks (MLPs and CNNs) and small-scale datasets (MNIST and CIFAR), see lines 33-36. In this work, we focused on popular backbone models for transfer learning. As we report in lines 494-497, we attempted our method on larger CNNs (e.g., VGG), and it wasn’t as successful. Reconstructing data from Transformer-based backbones proved more successful. In this respect, it is important to note that transformer-based backbones are very popular for feature extraction for transfer learning tasks in practice, and we believe that reconstructing data from transformer-based feature extractors is an important contribution on its own. We emphasize that our work is the first to reconstruct training images from real-world networks, rather than small-scale and unrealistic models as in previous works. We agree that exploring data reconstruction from other backbones is an important research direction, but beyond the scope of this paper.

• Thank you for pointing this out. Please note that all previous works on data reconstruction from trained models were done on much lower-resolution datasets such as MNIST (28x28) or CIFAR (32x32). Our work is the first to reconstruct training images from higher-resolution images (224x224), comparable to Imagenet-size images. Additionally, note that most standard backbones (like the ones we use), are trained at this resolution, and that typically, input images are first resized (and center-cropped) to this same resolution (224 pixels). Thus, this is a standard resolution. We agree that data reconstruction from higher resolution models is important, but this is beyond the scope of our paper.

• We make a great effort to establish a quantitative method for comparing reconstruction quality. We kindly refer the reviewer to the “Quantitative Evaluation” paragraph in lines 346-400. In particular, we compare the reconstruction quality between 6 different metrics (Figure 5). We additionally compare to activation maximization method (Appendix A.10) which is applicable in our setting. We emphasize that since this is the first work to successfully reconstruct training data from higher resolution images than CIFAR, it is difficult to compare to other methods, as there are none besides activation maximization. The reviewer’s suggestion is very good, we attempted this but it wasn’t successful. A possible reason may be, that quite surprisingly, the output of the network on reconstructed images can be very different from network output on corresponding original images, even though they are visually similar. There seems to be an inherent scaling issue in the reconstruction optimization. It is an interesting direction for future research.

• Thank you for your comment. We will resize Figure 9 to fit the width constraints. If there are other issues with the presentation we would be happy to fix them.

Questions:

• The KKT-based reconstruction is known to be successful for different types of networks (see [Haim et al. 2022], [Buzaglo et al. 2023]). Our method works for networks that can be “inverted” similarly to CLIP/DINO-ViT/ViT, hence we currently don’t know how to extend it directly to ResNets or other types of networks. This is an active field of research that we also work on, although it seems a more difficult task.

• As we mentioned, although it would be interesting to test even higher-resolution images, this is currently beyond the scope of our paper. The resolution 224X224 is used in most standard backbones (like the ones we use), and it is the Imagenet resolution. We think it is surprising by itself that it is possible to reconstruct images beyond a resolution of 32x32, which is the highest resolution achieved by previous works in the field. (also see our answer to weaknesses (2)).

• Please see our answer to weakness (3).

We thank the reviewer for the feedback.

• Thank you for pointing this out. In this work, we focus on models trained on embeddings of common backbone models, where the backbone models’ parameters are fixed (the so-called “feature-extractor” approach for transfer learning). We believe that this approach, while limited, is still the most common training approach used in practice. To support this, please see Kim et al. [1] who analyzed multiple papers on transfer learning for medical data and found that using backbones as feature-extractors is the most popular approach for transfer learning. We agree with the reviewer that exploring data reconstruction from models trained with other transfer learning approaches (e.g., fine-tuning backbone parameters) is of great importance and value to the community, however it is beyond the scope of this paper. Whether the current approach can be generalized to such settings is an interesting direction for future works.

• The main contribution is in demonstrating the vulnerability to training-data reconstruction of models that are trained in a transfer learning manner using embeddings of widely-used backbone models. This is a very popular approach for training classifiers on tasks with limited data, and exposing this vulnerability should be of major concern to practitioners. We provide an abundance of experimental evidence to support our claims. Another major contribution is the clustering-based reconstruction approach, which is very important in making such reconstruction attacks useful in practical settings. Note that many works in the field are using the training data in order to validate their attack. Our clustering-based approach alleviates this requirement.

• Right. We will fix it in the final version. Thanks.

[1] “Transfer learning for medical image classification: a literature review”, Kim et al. 2022

We thank the reviewer for the feedback.

• The main contribution is in demonstrating the vulnerability to training-data reconstruction of models that are trained in a transfer learning manner using embeddings of widely-used backbone models. This is a very popular approach for training classifiers on tasks with limited data, and exposing this vulnerability should be of major concern to practitioners. We provide an abundance of experimental evidence to support our claims. Another major contribution is the clustering-based reconstruction approach, which is very important in making such reconstruction attacks useful in practical settings. Note that many works in the field are using the training data in order to validate their attack. Our clustering-based approach alleviates this requirement.

• Thank you for pointing this out. Please note that all previous works on data reconstruction from trained models were done on much lower-resolution datasets such as MNIST (28x28) or CIFAR (32x32). Our work is the first to reconstruct training images from higher-resolution images (224x224), comparable to Imagenet-size images. Moreover, note that most standard backbones (like the ones we use) are trained at this resolution, and that typically, input images are first resized (and center-cropped) to this same resolution (224 pixels). Thus, this is a standard resolution. We agree that data reconstruction from higher-resolution models is important, but this is beyond the scope of our paper.

• We compare our method to two activation maximization methods (Appendix A.10) which are applicable in our setting. Since this is the first work to successfully reconstruct training data from higher resolution images than CIFAR, there are no other existing methods that we are aware of, and designed for direct comparison in such settings.

We thank the reviewer for the thorough and positive feedback.

Weaknesses:

• While we agree that some results look similar to class average, many are very similar to the original image itself (Fig. 3). We also think that such a comparison should be with respect to other techniques. In appendix A.10 we compare our results with two activation-maximization baselines on the same models. Note that the results from these techniques are far from being similar to any of the original images or their class representatives.

• Indeed, the choice of the backbone is significant to the success of the reconstruction, and especially for this reason we focused our experiments on very popular backbones for vision tasks (DINO-ViT, ViT and CLIP).

• The reviewer is correct in noticing that the inversion process is a bottleneck of our method. However, our approach is independent of the specific choice of the inversion technique, so that better inversion methods will certainly achieve better results. Additionally, encoder-based inversion techniques that we use for CLIP, are not so computationally expensive as the optimization-based techniques.

Questions:

• We compared our reconstruction method to two different activation maximization reconstruction schemes. The results are in Appendix A.10 and are significantly inferior to our reconstruction scheme. We additionally compared six different metrics to evaluate the reconstruction results (Figure 5), and in Figure 6, we show a high correlation between an image being close to the margin and having a high reconstruction score, which is predicted by the theory. Given that this is the first work to reconstruct images with a resolution higher than CIFAR from trained classifiers, activation maximization remains the primary method available for comparison in this context.

• Regarding diffusion-based generative models, works that focus on data reconstruction from such modes (e.g., Carlini et al. 2023) have a very different approach. They are also highly dependent on knowing the training data for their evaluation (e.g., specifically targeting highly duplicated samples in the training set, that are searched for prior to the attack, and then using their text prompts from the training data in order to generate image samples). While such works are of prime importance, our submission focuses on classifier models and emphasizes privacy risks in transfer learning tasks, which are widely used in practice.