Where have you been? A Study of Privacy Risk for Point-of-Interest Recommendation

Dear Reviewer KUtq,

We are grateful for the time you took to review our paper and for your valuable feedback. We would like to present our explanations and responses to your concerns below!

W1 Only Evalutes the Privacy Risk in A Controlled Setting But Not Real-World Scenarios
Our attack is performed on real-world data sets collected by leading social media companies. Since the real-world POI recommendation models are also trained on a similar data source, we believe that our attack on the current dataset should be able to simulate realistic scenarios where the attack might happen.

Also, as mentioned in Section 2.2, our attacks do have real-world effects, majorly in two cases where the adversary should have the abilities mentioned in our paper:

(1) In a scenario mentioned by service providers [1,2,3], user data is collected and used to train models, which are then provided as an API to third parties for application development. In this context, a malicious third party might gain access to the confidence scores from the model's output

(2) In a second scenario, a situation arises on the service provider side during the retention period of the training data expires. Still, the model owner retains the model, and an adversary (e.g., a malicious insider of location service providers) can aim to use the model to perform our attack aiming to infer information about the data that has been deleted.

As discussed in the “Limitation and Future Work”, Section 6, we also plan to adapt our attack method to some public real-world services with more limitations (i.e. Map APIs which might only provide the predicted label) using methods like label-only attacks.

W2 Comparison to Privacy Risks Associated With Ride-sharing or Food-delivery Apps
The prior ride-sharing and food-delivery problems mostly lie in solving the different problems of different privacy stakeholders compared to the POI Recommendation Models.

Firstly, prior ride-sharing and food-delivery research targets the following problems:

(1) The privacy of company-owned data. For example, Lyft and Uber can use the spatial distribution information of their instrumented fleets for better placement and routing [4].

(2) Privacy attacks targeting passengers or companies' service providers, such as drivers. For instance, attackers could steal continuous trajectories and analyze behaviors. However, our paper focuses on attacking a different data format: the user's POIs in location formats, which can be combined into sparse trip trajectories closely related to the user's daily life.

Secondly, the defense mechanisms are different. The studies of Ride-sharing or Food-delivery primarily focus on the data aggregation process, with most relying on secure system design utilizing methods like cryptographic approaches or differential privacy. Our attacks, however, target the privacy leakage of deep learning models, representing a new attack surface. The defense mechanisms for ride-sharing or food delivery are not suitable to defend against our attack.

Q1 Explanation for Defense Mechanism Not Working and Insights Into More Effective Defenses
Thanks for asking this question. This is a great question. We have listed the results of commonly used defense mechanisms in Appendix E. These defense mechanisms(e.g. differential privacy), often result in significant drops in utility, primarily due to the nature of the task. The reason is that the utility of existing POI recommendation models heavily depends on memorizing user-specific trajectory patterns from limited user-level data, which lacks sufficient semantic information as in CV/NLP. However, defense mechanisms like differential privacy can diminish the level of memorization.

We believe that to solve this problem, we need a better-designed scheme that can help with generalizing the POI recommendation model. For example, more information in the dataset, such as POI categories or trip purposes, should be included, and this information should be better utilized. Also, we believe that using a pre-trained model on a public dataset and fine-tuning it on a smaller private dataset might help solve this problem. This approach has been successfully used in other tasks in CV [6,7] and NLP [8,9]. This can be a potential future direction, and we plan to work on this in the future.

Dear Reviewer sUA6,

We thank you for your valuable feedback and for finding our work meaningful! We would like to respond to your concerns and improve this paper based on your suggestions.

W1 Effect of Larger Dataset
Thank you for pointing this out. We believe the dataset size does not affect our key insights, as our attacks are derived primarily from the task's inherent characteristics. For example, POIs frequently visited by a user are typically more unique to that user (e.g., home address). These locations are always more susceptible to our attack since they are well memorized by the models as shown in Figure 4.

To confirm this, we conducted additional experiments using a larger FourSquare dataset from Tokyo, containing about 4x more check-ins than the NYC dataset. Our LocExtract and LocMIA attacks on models trained on this dataset are still effective, showing that our attack is agnostic to the dataset scale. Due to the time limitation in the rebuttal period, we have not finished the experiment on the trajectory-level attacks but we will add them to our paper when finished.

LocExtract Result:

LocMIA Result:

W2 More Descriptions of Dataset
Thanks for pointing out this. We have included some dataset-related information in Table 3 in the Appendix, and we will make sure we highlight this information further. Our attack uses real-world datasets that are collected from NYC social media users.

W3 Attack Comparison with Other Types of Data/Models
Thanks for pointing this out! We will add a table for a comparison of MIAs here, as the data extraction attack we proposed is specific to POI recommendation models. As mentioned in [1], LiRA can achieve an 8.4% TPR @0.1%FPR on image classification models trained on the cipher-10 dataset. In our case, the attack achieved a 20+% TPR @0.1%FPR for LocMIA on all models trained on the FourSqure dataset, but it’s relatively lower for the Gowalla dataset with a 1+% TPR @0.1%FPR. As demonstrated in [1], the difference in the attack success rate can be attributed to the level of memorization in the model.

However, it should be noted that the current POI recommendation model utility still heavily relies on the memorization of the historical visit of a user, which makes the attack relatively easier and leaves the protection of POI-recommendation models more difficult, as we will discuss in the next question.

Q1 Reason for the Utility Privacy Trade-off and Potential Practical Solution
This is a great question/ We have listed the results of commonly used defense mechanisms in Appendix E. These defense mechanisms(e.g. differential privacy), often result in significant drops in utility, primarily due to the nature of the task. The reason is that the utility of existing POI recommendation models heavily depends on memorizing user-specific trajectory patterns from limited user-level data, which lacks sufficient semantic information as in CV/NLP. However, defense mechanisms like differential privacy can diminish the level of memorization.

We believe that to solve this problem, we need a better-designed scheme that can help with generalizing the POI recommendation model. For example, more information in the dataset, such as POI categories or trip purposes, should be included, and this information should be better utilized. Also, we believe that using a pre-trained model on a public dataset and fine-tuning it on a smaller private dataset might help solve this problem. This approach has been successfully used in other tasks in CV [6,7] and NLP [8,9]. This can be a potential future direction, and we plan to work on this in the future.

Q2 Explanation Why More Total Check-ins Help Protect a User From MIA
Thanks for asking! Sensitivity in DP measures the maximum difference between samples in a dataset. When this measurement is at the user level, a user with more check-ins can be more easily distinguished from others, as suggested in [2]. However, this kind of user-level inference attack might not be suitable for this task. The POI recommendation model takes the user ID as input. If such a user does not exist in the dataset, there will be no query results. Rather than focusing on a user-level inference attack, our attack aims to infer the membership of individual POIs or trajectories. In this context, if a user has visited many locations, the model may have lower confidence in predicting a single POI, leading to less distinction in confidence scores between members and non-members. This scenario can be likened to the idea that hiding a location among a large number of locations is easier than concealing it among a few. Consequently, the more locations a user has visited, the lower the performance of the attack.

Q3 Feasibility of DP Synthetic Data for POI Recommendation
Synthetic data with DP guarantee is a direction that is worth more investigation. There are some existing studies that discuss the use of synthetic data to protect mobility information, with a primary focus on safeguarding continuous trajectories. For instance, [3] explores the use of differentially private synthetic data to protect POIs. However, they do not address the utility and performance of POI recommendation tasks, as their main target is the private release of data.

Our paper mostly focuses on measuring the privacy leakage from the POI recommendation models. However, Synthetic data generation might require different threat models and attack settings, Thus, we do not include them in our paper. Note that privacy attack for synthetic data is a promising direction that needs independent work to investigate [4, 5]. As a future work, it is interesting to see if the privacy attacks in these works can be applied to mobility data synthetic models like [3].

References:
[1] Carlini et al. Membership Inference Attacks From First Principles.
[2] Montjoye et al. Unique in the crowd: The privacy bounds of human mobility.
[3] Rao et al. LSTM-TrajGAN: A Deep Learning Approach to Trajectory Privacy Protection.
[4] Breugel et al. Membership Inference Attacks against Synthetic Data through Overfitting Detection.
[5] Hyeong et al. An Empirical Study on the Membership Inference Attack against Tabular Data Synthesis Models.
[6] Yu et al. ViP: A Differentially Private Foundation Model for Computer Vision.
[7] Sander et al. Tan without a burn: Scaling laws of dp-sgd.
[8] Anil et al. Large-scale differentially private bert.
[9] Yu et al. Differentially private fine-tuning of language models.

Q8 Explanation of Fluctuation in Figure 5(b)
We believe that the fluctuation in Figure 5(b) can be attributed to two main reasons.

(1) We hypothesize that when the trajectory is very short, it may be more easily influenced by other factors, such as increased location sharing. Additionally, as mentioned in our paper, many other characteristics can impact the performance of the attack. So they can also contribute to the fluctuations.

(2) When the trajectory is longer, it is more difficult to be memorized by the model. Since our attack considers all location points in the trajectory, it is more likely some locations are less well memorized than others and cause a drop in attack success rate. This also explains why the overall trend is that longer trajectories are less vulnerable to our MIA, and a similar result can also be observed in Figure 17, where we conduct a study from the attack parameter selections.

Q9 Different Selection of “topk-accuracy” and ”topk-asr”
Thanks for pointing out this notation confusion. The topk-accuracy of (1,5,10) on page 3 is in evaluating the performance of the POI recommendation task itself, whereas in the later part of the paper, we also use the same setting (and mostly k=10) to evaluate the utility of the model with/without protection.

However, the result in Figure 1 is actually topk-ASR for the data extraction attacks, where a smaller k makes more sense as it represents the reliability of the attack. That’s why we use k=1/3/5 to evaluate the attack results. We will state this clearly in the next version of the paper.

References:
[1] Liu et al. ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models.
[2] Carlini et al. Membership Inference Attacks From First Principles.
[3] https://gdpr.eu/article-5-how-to-process-personal-data/
[4] https://gdpr-info.eu/art-25-gdpr/ [5] Andrés et al. Geo-indistinguishability: Differential privacy for location-based systems.
[6] Bao et al. Successive Point-of-Interest Recommendation With Personalized Local Differential Privacy.
[7] Xu et al. An efficient privacy-preserving point-of-interest recommendation model based on local differential privacy.
[8] Chen et al. Curriculum Meta-Learning for Next POI Recommendation.
[9] Liu et al. STGIN: Spatial-Temporal Graph Interaction Network for Large-scale POI Recommendation.
[10] Luo et al. Timestamps as Prompts for Geography-Aware Location Recommendation.
[11] Yang et al. GETNext: Trajectory Flow Map Enhanced Transformer for Next POI Recommendation.

Dear Reviewer 1v9T,

We thank you for taking the time to review our paper and provide valuable feedback and questions. We would like to present our explanations below and hope they can address your concerns.

W3 Experiment and Insights Are Not Surprising
We are willing to adjust the problems or explain our experiments in more detail. Can you please provide more suggestions on the experiments that could further improve the experiment?

W4 Typos In The Paper
Thank you for pointing out the typos. We will fix them in the next version.

Q1\W2 Need Formally Define Sensitive Information and Privacy Guarantee
We have included the notations and definitions of the objectives for each attack in Appendix Table 2, accompanied by their respective mathematical definitions. For your convenience, the corresponding definition table is also attached below for reference.

Our definitions of sensitive information in our attack suite are similar to previous works studying privacy risks on general machine learning models [1,2]. We appreciate it if you could provide additional suggestions and clarifications on improving the definition of sensitive information.

Q2 Need Defense-Related Literature Survey
We have listed the attack-related works in section 5 and defense-related works in appendix F.

As mentioned in Section 5, prior research primarily focused on exploiting side channels such as social relationships and historical trajectories to extract sensitive information during the data aggregation and release processes. However, these studies have not yet concentrated on potential leakages through deep neural networks. Given that more recent LBS relies on deep neural networks, which present a new attack surface for adversaries seeking to access private user information, these earlier attack studies are insufficient. We are the first to investigate privacy leakages in POI recommendation models based on deep neural networks. Our work highlights the need for a privacy-preserving ML-based POI recommendation system in both industry and academia.

As for defense, prior works primarily focused on data aggregation and release processes tailored to the attack mentioned above. Since our paper focuses on evaluating the privacy leakage associated with the model, we only focus on the defense mechanisms during the training process, which is widely ignored in prior works. We also will discuss more defenses, e.g., LDP) in the related work section.

We also appreciate it if you could identify some missing related work, and we will cite and compare them accordingly.

Q3/W1 GDPR and Attack Motivation
We would like to emphasize that the threat model of the paper is well-motivated, and the privacy study of the work is urgently needed, as required by GDPR.

First of all, GDPR does not mention requirements for using specific protection mechanism mechanisms like differential privacy in collecting user data like location data. GDPR only requires “the collected content is needed, minimized, and stored safely” in articles #5 and #25 [3,4]. Since the potential threats in POI recommendation models have not been well studied, companies currently have not adopted adequate protection mechanisms when receiving the data [8,9]. Even if there are protection mechanisms like differential privacy on training data, existing works lacks measurement study to quantify whether those data protection mechanisms are properly applied to the data. Our work could bridge this gap by using our proposed attack suite to perform such kind of measurement.

Moreover, no prior work has studied how POI recommendation models leak sensitive information about users’ data, and our work is the first one to study this problem. This is an important problem since POI recommendation models could unintentionally leak user’s sensitive information.

Dear Reviewer 1v9T,

Thank you very much for taking the time to review our submission. We deeply appreciate your thorough feedback and evaluation of our work. In our rebuttal response, we have carefully considered your comments. We would be grateful for your acknowledgment of our responses and your feedback on whether they address your concerns. We are happy to engage in further discussions if needed. Once again, thank you for your invaluable input and attention to our work.

Dear Reviewer iRwY,

We thank you for taking the time to review our paper and for acknowledging the importance of our research problem. We attach our responses to the concerns below and hope to adequately address them.

W1 Baselines Missing
Thanks for the suggestions! We believe that the existing inference attacks can work, but we have not included them in the paper for the following reasons:

(1) Since our paper primarily focuses on evaluating the privacy risks in POI recommendation models. Thus, we select the SOTA attack – LiRA [2], as it has been shown to outperform other MIAs, including [1].

(2) Our main contribution is in adapting the existing methods to the POI context. Our proposed mechanism should also work across different attack approaches like [1] as well, as they both utilize similar techniques like shadow model training.

W2 Qualitative Result
We will add some qualitative result analysis on the data extraction attack and corresponding visualization (e.g., map visualization) to help readers better understand the analyses of the attack. We also appreciate it if you could provide suggestions on the qualitative study that we can conduct.

W3 Threat Model and Confidence Scores
Thanks for asking. We define the corresponding threat model and realistic cases in section 2.2 (Adversary Knowledge). We highlight the scenarios below:

(1) Instead of focusing on publicly available models like Google Maps, which might only provide the final predicted label, we are targeting the scenarios of service providers [3,4,5], who have mentioned in their service that collected user data and trained models, then providing these trained models as services to 3rd party to build applications. Under this setting, a malicious 3rd party might have the confidence scores from the model output.

(2) Also, we have mentioned another scenario where the retention period of the training data expires. Still, the model owner keeps the model, and an adversary (e.g., a malicious insider of location service providers) can aim to use the model to perform our attack to infer about the deleted data.

References:
[1] Shokri et al. Membership Inference Attacks against Machine Learning Models
[2] Carlini et al. Membership Inference Attacks From First Principles
[3] Chen et al. Curriculum Meta-Learning for Next POI Recommendation
[4] Liu et al. STGIN: Spatial-Temporal Graph Interaction Network for Large-scale POI Recommendation
[5] https://www.localogy.com/2021/03/foursquares-power-play-continues-with-relaunched-places-and-new-api/