To Reviewer 1W2d

Q1: What would be the main difference between the proposed method and the approach outlined in [1]?

We apologise for missing this important work in our paper. We will take your advice to cite and discuss this paper. Our attacks are highly different from this paper from the following perspectives.

(1) Difference in the Threat Models. The method proposed in [1] is to work in the circumstances where the attacker is able to obtain the logits of the model, whereas our method is exclusively designed for the label-only situations.

(2) Difference in the loss function. The difference in the threat model also leads to the difference in the loss function design. In [1], since the attacker can get the predicted logits, the main purpose of the loss function is to ‘optimise $x$ so that IN shadow models have low losses on $x$ and OUT models have high losses on $x$’. It directly minimises the logits of the out models by using the loss term $-\log(1-f_\theta(x)_y)$ where $f_\theta(x)_y$ stands for the logits corresponding to the ground truth $y$, which will result in a smaller logits, i.e., smaller $f_\theta(x)_y$.

In contrast, our attacker can only get the hard labels. Therefore, the goal of our loss function is to optimise $x$ so that it can be correctly classified by IN models while being mis-classified by the OUT models. In this case, it is problematic to just enlarge the variance in the logits, confidence scores or the loss values, as it may not be enough to cause the differences in the predicted labels of the query samples between the in and out models. To solve this problem, we propose to use $CE(f_{out}^i(x),l_i')$, where $l_i'$ is the 'nearest' class of the target sample $x$ other than its ground truth. Unlike $-\log(1-f_\theta(x)_y)$, the $CE(f_{out}^i(x),l_i')$ term tends to ensure the label difference. When minimising the cross-entropy, it enlarges the logits of target label l_i’ at the mean time minimising the other logits to guarantee the predicted-label-level differences. Moreover, to carefully choose the target label $l_i’$ helps us to push the query sample into the ‘improvement area’, i.e. the most sensitive part of the decision boundary towards the membership difference between the in and out models, therefore further enhancing the attack. (We’ve demonstrated the effectiveness of choosing target labels in the ablation study of the manual.)

(3) Difference in the attack effectiveness To further see the differences, we migrate the method in [1] to the hard label scenario, and measure the attack effectiveness. We adopt the method in [1] for the query-sample-generation (Algorithm 1 in our paper), while the rest parts are the same. We perform evaluations on CIFAR10 (2,500 samples) using CNN7. The rest of the settings are the same as the experiments in the paper. The results are shown in the table below. It is clear that our methods are much more effective than [1] in terms of label-only MIA.

Q2: It's encouraging that the crafted query is transferable. However, if the training algorithm of the target model differs from that of the shadow model, such as having different learning rates or optimizers, would the attack still retain its potency?

We conduct experiments using victim models trained with various settings that are different from the shadow models. Specifically, we test different optimizers, batch sizes, and learning rate. All the experiments are conducted using the same setting in the ablation studies. The results are shown below.

We observe that the different training details such as batch size and optimizer have slight impacts on the effectiveness of our methods. The crafted query samples from YOQO have high transferability.

Reference:

[1] Wen, Y., Bansal, A., Kazemi, H., Borgnia, E., Goldblum, M., Geiping, J., & Goldstein, T. (2022). Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries. arXiv preprint arXiv:2210.10750.

To Reviewer 1cwK

Q1: The use of accuracy as the primary evaluation metric is questioned, as it may not reflect worst-case performance in MIA.

We appreciate your suggestions on how to show the worst-case performance in MIA, which is really inspiring. Following your suggestions, we use the distance provided by boundary attack [1] as the metric to pick the worst-case samples, which directly measures the distance (L0, L1, L2, L $\infty$) in the space of the target samples. W e calculate the distance on 5 victim models and take the averages as the distance. Then we select the top 10% samples (200 samples) with the largest distance. The results are shown below:

We believe this result can reflect the worst-case situation to some extent. We observe that our methods tend to have better performance than the boundary attack. This indicates that YOQO can be more effective to infer the membership of the worst-case samples which the boundary attack is hard to deal with. To strengthen the solicity of our work, we will add the discussions in the revision.

Q2: The target model is trained with a relatively small dataset of 2,500 samples, which could potentially result in severe overfitting. I would like to see the authors conduct experiments on well-generalised models, involving a larger training dataset to mitigate the risk of overfitting.

(1)We would like to clarify that we varied the size of the training dataset from 1,500 to 10,000 in the paper to show how the overfitting would affect the attack performance. The small dataset of 2,500 samples is only used for defence evaluation. As shown in Fig.2, the smallest gap between the test and train set is around 24% (training set accuracy = 98%, test set accuracy = 74%)

(2)Following your suggestions, we further do the experiments using bigger datasets (25,000 samples of CIFAR10 and 75,000 samples of Tiny-ImageNet) to reduce the overfitting. To make the target model more well-generalised, we also conduct experiments with models pretrained on Imagenet. The results are shown in the table below. In all cases, YOQO retains its comparable performance as the boundary attack.

(3)Additionally, to explore the impact of overfitting on attack effectiveness, we conduct the experiments using L2 regulation to further reduce the overfitting. The results are shown below. Our attacks are still comparable with SOTA attacks but with much more efficiency (query only once).

We will add all the discussion and evaluation results of overfitting in the revised paper.

Q3: Whether adding a weight term to Equation 2 could achieve the same result, that is, turn the output of in models to [0, $\lambda$].

Indeed, having a weight term $\lambda$ can scale the output of in models to [0, $\lambda$], which may ameliorate the imbalance between the two terms. However, the real problem lies in the range of $CE(\frac{\lambda \cdot f_{in}}{f_{out}})$. For any given $f_{in}$, the loss $CE(\frac{\lambda \cdot f_{in}}{f_{out}}, l)$ ranges in $(-\infty, CE(\lambda \cdot f_{in}, l)]$, which means for any threshold $t$, we can theoretically find a $f_{out}$, so that $CE(\frac{\lambda \cdot f_{in}}{f_{out}})<t$, indicating the algorithm is not convergent. This means we cannot simply set a threshold on the total loss to decide when the algorithm should stop. In this case, we need to carefully choose the total iterations of the optimization as well as the $\lambda$ to ensure the acceptable inference accuracy, which is not that practical in the real word setting, and also means it is a sub-optimal choice (for the best $\lambda$ and iteration numbers can vary from samples to samples). Below we do the experiments to demonstrate the our opinion. We varies the $\lambda$ from 1 to 16. The results indicate that the performance in this case is more dependent on the $\lambda$, and is less effective in comparison with the original loss as in Fig.6 shows in the paper, which used the same settings of this experiment, i.e., 2,500 samples on CIFAR10.

Thank you for the suggestions to enhance the robustness and rigor of our work. Please feel free to reach out if any further concerns arise.

To Reviewer bo89: (2/2)

Q4: Utility of the attack: I could not understand from the current evaluations how the attack will perform in the real world. The dataset sizes are quite small which is seldom the case nowadays, unless the model is fine-tuned using a small dataset. Can authors provide results for some real-world settings? Some suggestions: 1) use larger datasets 2) fine-tune a model pre-trained on large datasets?

Thank you for the suggestions. We follow them to evaluate our attacks on these new settings. Particularly, 1) we use larger training datasets with higher resolutions and more samples (75k Tiny-ImageNet, 25k CIFAR-10); 2) we consider both training from scratch and fine-tuning a public model pre-trained on ImageNet. The inference accuracy of different attacks are shown in the following table. We can observe that our attacks maintain comparable accuracy as SOTA attacks, but much more efficient (only query once). This conclusion is consistent with the evaluations in the paper, demonstrating the practicality and utility of our attacks in a wider set of configurations.

References:

[1] Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In International conference on machine learning, pp. 1964–1974. PMLR, 2021.

To Reviewer bo89: (1/2)

Q1: Although the experimental setup considered is the common setup in most prior works, it contains mostly small datasets with very small training dataset sizes. Can authors perform experiments on larger datasets, e.g., Imagenet?

Response:

Thanks for the great suggestion. The size of the training dataset used in the current manuscript ranges from 1,500 to 10,000, with various data types (e.g., images, tables). These are indeed common setups in existing MIA works. Following your suggestion, we'd like to extend the evaluation to larger datasets. Performing evaluations on Imagenet requires quite a long time, especially the online attack, which will exceed the response deadline. Alternatively, we conduct extra experiments on Tiny-Imagenet, a subset of Imagenet which contains over 100k images of 64 $\times$ 64 size belonging to 200 categories. The evaluation results can be found in our response to your Q4. Our method remains effective in larger datasets. The results can be found in the second part of the response.

Q2: I could not figure out the training recipe from the main paper. In particular, does the training ensure that the models don’t overfit to the training data, e.g., using L2 regulation or any other common regulation techniques? I feel the criterion that training goes on until a model has >98% accuracy on training data may lead to overfitting that will lead to unnecessarily stronger MIAs.

Response: (1)When training the models, we indeed used the L2 regulations and set the regulation parameter $\lambda$ to be 5e-04. The influence of L2 regulation may not be strong enough to entirely prevent overfitting. Yet it is a common setting for training models on the datasets we use, and we believe the overfitting is reasonable, the same as other MIA works.

To demonstrate the effectiveness of YOQO on less overfitting models, we further perform evaluations on models trained with stronger L2 regulations. Specifically, we use CNN7 as both the shadow and victim model, over 2,500 CIFAR-10 images as the training set. The results are shown below. As the weight decay $\lambda$ increases, the effect of the L2 regulations get stronger, resulting in performance degradations in all of the attacks. Nevertheless, YOQO still keeps comparable performance as the Boundary attack. Additionally we also notice that strong L2 regulation makes the training very unstable, and also causes performance degradations on the test accuracy.

(2) For the condition of training termination, we would like to clarify that we did not take the training set accuracy as the criterion to stop training. Instead, for each task and dataset, we stop the training when the model achieves the best performance on the validation set. Such models will also get >98% accuracy on the training set. Sorry for the confusion, and we will add all the training details in the revised paper.

(3)Additionally, the models evaluated in our paper are actually less overfitting in comparison with the Boundary attack paper [1]. For instance, using 2,500 samples from CIFAR10 for training, the ASR of gap attack in our work is around 70%, while that of [1] is around 75% (Fig.1 of [1]). As the gap attack is correlated to the gap between the performances on the train and test sets, a lower ASR in gap attack means our models are less overfitting in comparison with that being tested in the Boundary attack paper [1].

Q3: On the same lines as the above comment, can authors add a few more details of various dataset sizes used in training with/without defences, e.g., adversarial regulation? Also can they add details of training procedure? These are important given that MIA efficacy is greatly affected by these factors.

Response: Thank you for pointing this out. We are sorry for missing the training details due to the page limit. In our paper, for all the defence evaluations, we use 2,500 CIFAR-10 samples as the training dataset. For the attack effectiveness evaluations, we use different datasets with different numbers of samples from 1,500 to 10,000. These details can be found at the beginning of Sections 4.2 and 4.3. Our training settings are shown below:

We will clarify those details in the revised manuscript.

We thank all the reviewers again for your valuable feedback!

We further adjust the manuscript following the suggestions of Reviewer bo89.

• section (4.2 table.3): adding model architectures.

• section (4.2 table.2): adding the size of dataset in the caption.

all the parts that are different from the original manusacript are in blue for identifications.

We hope these improvements and our responses can address your concern. We sincerely appreciate your guidance and support