Mobility-LLM: Learning Visiting Intentions and Travel Preference from Human Mobility Data with Large Language Models

Thank you for your thoughtful and constructive comments.

[W1&W2&Q1]

We design the VIMN module to reprogram the check-in sequences. After passing through the VIMN module, the check-in sequences are aligned to the natural language semantic space, producing the resulting $\mathbf{h}_i$. This ensures that the two representation vectors sent to the HTPP module are in the same semantic space. We believe that in the same semantic space, vectors that are close in distance have similar semantics. Therefore, if the cosine similarity calculated by the score-function is high, it indicates that $\mathbf{h}_i$ itself has semantic relevance in the semantic representation space. In summary, the key to evaluating the semantic similarity in urban mobility tasks lies in the design of the VIMN reprogramming module, with cos-similarity only serving to compare the similarity of vectors in the same semantic space.

[W2]

While the PPEL and HTPP modules leverage existing technologies from a technical perspective, our design principles are innovative and introduced for the first time in tasks related to human activities. These modules are integral, as they employ LLM's word token embeddings for both check-in sequence embeddings and prompt tokens, achieving semantic alignment and integration with other modules. Ablation studies have shown that these modules significantly enhance performance, thereby validating the efficacy and rationality of our design.

[Q2]

We considered the scheme you originally mentioned. However, listing all combinations creates a very large space, reducing the model's efficiency. Manually screening reasonable combinations will cost a huge amount of human resources and may fail to cover all possibilities due to the diversity of human activity data and the many situations that can arise. Additionally, case studies show that the existing scheme is superior in efficiency and accuracy compared to the one you mentioned.

I am disappointed with the author's rebuttal. The author did not sufficiently address my questions; the responses lacked clarity and evaded the main issues. For instance, I don't understand the purpose of the combined response to W1&W2Q1. Your answer is completely off-topic, and I feel that W2's question should correspond to W3. Specifically, my questions are:

1. I have concerns about the robustness of the human travel preference prompt (HTPP) module. For example, your HTPP module includes several critical parameters, such as D. Why is the domain set to 3? Is this a limitation of the dataset, an arbitrary definition, or derived from experiments? I am unclear. What is the reason for setting m to 16? Even if the author did not mention this in the paper, it should be adequately addressed in the rebuttal. Unfortunately, I did not find any relevant answers.

2. My question is why you chose cosine similarity instead of other methods like Euclidean distance or Manhattan distance. These could all be discussed. Of course, I know that cosine similarity can measure the similarity of vectors in a unified semantic space, but what are its specific advantages? Did you determine through experiments that cosine similarity is better? Can't these topics be further discussed?

3. Your response claims that the existing solution is superior to the idea I proposed in Q2. The original text states, "We provide three case studies to visualize the improvements brought by HTPP and VIMN." I do not understand the relevance of the cases you provided to my question. I need a clearer explanation, such as comparing my question's definition with your case studies. Your response only added to my confusion.

Other reviewers have also provided insights from different perspectives, for example:

1. Reviewer NycQ believes that the performance improvement is not significant enough, which is indeed a problem. Your rebuttal mentioned that the t-value between Mobility-LLM and the best baseline is very high. However, I think this is because both models are very stable, and changing the random seed does not overly affect the model's performance, leading to a low standard deviation and a high t-value. This is meaningless. The core issue is that your improvement over the best baseline is only 0.4%. While such an improvement might be acceptable for other deep learning papers, this paper is based on LLM, requiring better GPU, more computing time, and potential API costs, making the slight performance improvement questionable.

2. For example, the classic baseline Graph-Flashback [1] for next location prediction shows more than double the performance improvement on Gowalla compared to DeepMove, and 4% higher on Foursquare. Why didn't you compare with some classic baselines, or if your performance improvement could also reach this level, it would be more reasonable. DeepMove is only a baseline published in 2018, but according to Table 1, your model only shows about a 1% improvement over DeepMove on Gowalla and Foursquare.

3. The authors claim that their convergence speed is faster than other baselines, but I did not see any relevant evidence. If you are referring to fewer epochs, I think this is unreasonable. The time each model consumes per epoch is completely inconsistent. The speed of convergence should consider the overall training time. What I observed is that Mobility-LLM consumes more memory, requires more training time, and inference time. The increased computational cost only brings minimal performance gains.

4. Reviewer Mti9 considers your Figure 1 to be rather cluttered. Why didn't the authors use the global response to upload a PDF showing the improved figure? After your adjustments, we are not clear about what the final version will look like.

5. Additionally, CACSR was not discussed. Why didn't you present a relevant discussion in the rebuttal?

Overall, although the authors initially received an overall positive comment, I do not see the authors taking the reviewers' questions seriously in the rebuttal, and the evasive answers have deepened my doubts. It is undeniable that the completion of this paper, including the production of charts and experimental results, is very high. However, the vague explanations of many details, the additional computational costs, and the insufficiently superior performance lead me to lean towards rejecting this paper, despite the positive attitude of other reviewers.

[1] Rao, Xuan, et al. "Graph-flashback network for next location recommendation." Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining. 2022.

Thank you for your further reply, which basically solved my concerns (Yours part). Please also answer my comments on other questions.

【Other-Q1】 Taking Table 1 as an example, for acc@1, can I understand that the performance of deepmove under the default settings is only 10.51/1.37=7.67?

【Other-Q2】 What about baseline GETNext [1], which supports variable-length input? This is the most popular baseline for next location prediction in the past two years.

[1] Yang, Song, Jiamou Liu, and Kaiqi Zhao. "GETNext: trajectory flow map enhanced transformer for next POI recommendation." Proceedings of the 45th International ACM SIGIR Conference on research and development in information retrieval. 2022.

【Other-Q3】 Mobility-LLM (TinyLlama)-Patience 3 11.3G 2.96h 10 The results in this row seem to indicate that the computational efficiency of the model is slightly weaker than that of ReMVC, and the memory it occupies is much larger than that of ReMVC, but the performance improvement is very significant, right?

【Other-Q4】 In the main figure, LLM (Partially-Frozen+LoRA) is missing the number (d) In sub-figure (a), the order of the attention module is v, k, q, and in sub-figure (d) the order is q, k, v. It is recommended to unify the order. In addition, why are q and k in sub-figure (d) below the weight matrix, while v is part of the weight matrix?

Thank you for your detailed feedback. We appreciate the concerns raised and would like to provide a comprehensive response that emphasizes the significance and contributions of our work:

1. Innovative Use of Reprogramming Techniques

   Our model represents the first application of reprogramming techniques (e.g. VIMN and HTPP Module) to Large Language Models (LLMs) specifically for understanding and predicting human activity sequences from check-in data. This approach is novel and transformative within the field of check-in sequence prediction, allowing LLMs—traditionally used for natural language processing—to be effectively adapted for this unique task. By reprogramming LLMs for the prediction of check-in sequences, we are pioneering a new direction in the field, demonstrating that these models can be utilized to extract and interpret complex human activity patterns, which were previously challenging to model with traditional methods.

2. Semantic Understanding and Accurate Predictions

   Our experiments clearly show that our reprogramming design enables the LLM to understand the semantic information embedded within human activity sequences. This understanding is crucial, as check-in data is not just about locations and times; it carries rich context about human intentions, behaviors, and routines. The model's ability to leverage this semantic understanding to make accurate predictions of future check-ins highlights a significant advancement in the field. It demonstrates that LLMs, when reprogrammed appropriately, can effectively capture and utilize the intricate patterns and meanings in human activity data, leading to more precise and contextually aware predictions.

3. Beyond Pure Performance Metrics

   We recognize the importance of performance metrics; however, the value of our work extends beyond marginal performance improvements. The primary contribution of our model lies in its innovative application of LLMs to check-in sequence prediction, introducing a new methodology that bridges the gap between language models and human activity prediction. Evaluating our work solely based on performance metrics does not fully capture the broader implications and future potential of this approach. We believe that the introduction of such a novel methodology provides a foundation for future advancements and improvements in the field, which may yield even greater performance gains as the techniques mature.

4. Model Size and Memory Requirements

   Regarding concerns about the model size, it's important to note that our model's memory footprint, which is under 12GB, is well within the capabilities of modern GPUs. With the widespread availability of GPUs with large memory capacities, our model is practical and accessible for both academic research and potential industry applications. Additionally, we have demonstrated through our experiments with the pythia-70M base model that our approach can achieve superior performance while requiring less memory and computational time compared to other baseline models. This showcases our model's efficiency and adaptability, ensuring it remains practical even in resource-constrained environments.

5. Robustness Across Multiple Datasets

   A key strength of our model is its robustness and stability across multiple datasets. In the context of check-in sequence prediction, where data variability is common, our model consistently performs well across different datasets. This generalization ability is crucial, as it indicates that our model is not overfitted to specific scenarios but is instead capable of adapting to various types of human activity data. The consistent improvements observed across four different datasets validate the effectiveness of our design and its potential for broader application in diverse real-world scenarios, such as personalized services, urban planning, and more.

6. Future Prospects and Model Versatility

   As LLMs continue to evolve, particularly with the trend towards smaller and more efficient models, our approach is poised to become even more effective. The ongoing advancements in LLM technology will allow our reprogrammed models to achieve better results with reduced computational resources. Our model is designed to be versatile and adaptable. It can be easily integrated with any large model available on huggingface, ensuring that our approach remains relevant and capable of benefiting from the latest developments in LLMs. This adaptability makes our model a future-proof solution for human activity prediction.

In conclusion, we believe that our work represents a pioneering effort in the field of check-in sequence prediction and human trajectory analysis. The innovative application of reprogramming techniques to LLMs on our tasks not only demonstrates the feasibility and effectiveness of this approach but also provides a robust and adaptable solution that can inspire and guide future research.

Thank you for your thoughtful and constructive comments.

[W1]

Thank you for pointing out these errors. We have made the corrections and checked other parts of the paper for similar issues.

[W2]

Compared to trajectory data (such as vehicle trajectories), check-in data are recorded more sparsely. Each point in a trajectory might record a passing-by, while each point in a check-in sequence corresponds to a purposeful visit to a POI. To achieve optimal performance on trajectory data, some modules of the proposed methods may need to be redesigned to fit the characteristics of trajectories.

[W3]

Thank you for your suggestions. We have adjusted the overall framework diagram to highlight key modules and provided detailed diagrams of each module as separate illustrations.

[W4]

Thank you for your suggestions. We will supplement the implementation settings for the few-shot scenario. In this scenario, we first divide the entire dataset into training, validation, and test sets in a 6:2:2 ratio based on the number of check-in sequences. Then, we only reduce the training set portion to 20%, 5%, and 1% of the original training set, while keeping the validation and test sets intact.

[W5]

Sorry for the oversight. We have added all the baseline models in the main text and appendix and included detailed discussions in the appendix.

[Q1]

The dimension of $\alpha$ is $(B,n,d)$, and the dimension of $\beta$ is $(B, 3, d)$. First, we concatenate along the second dimension, resulting in a dimension of $(B,n+3,d)$. Then, we average pool along the second dimension to obtain $(B,1,d)$. Finally, after a fully connected layer and softmax operation, we obtain the predicted user. Here, $B$ represents batch size, $n$ represents sequence length, and $d$ represents embedding dimension.

[Q2]

We would like to point out that JS divergence is more commonly used for tasks where the output is a sequence. Our model's prediction task only predicts a single future point, so we choose the current evaluation metrics, which are also widely used in baselines and most existing works.

Thank you for your thoughtful and constructive comments.

[W1]

The user embedding $U_i$ is an index-fetch embedding module implemented using the nn.Embedding module in PyTorch. It finds the corresponding embedding vector for a given user index.

[W2]

Sorry for the confusion. We want to clarify that these two expressions are actually equivalent in our implementation, and we will ensure consistency and standardize the terminology thanks to your suggestions. Below is a detailed explanation of why these two losses are equivalent in our case.

Cross-entropy loss is a method used to measure the difference between two probability distributions, commonly used in classification problems. It is often used to train neural networks, especially in classification tasks.

Assume we have a classification task where the true label distribution is $y$ and the predicted probability distribution is $\hat{y}$. The cross-entropy loss is defined as:

$H(y, \hat{y}) = - \sum_{i=1}^n y_i \log \hat{y}_i$

For a binary classification problem, the cross-entropy loss can be simplified to:

$H(y, \hat{y}) = - (y \log \hat{y} + (1 - y) \log (1 - \hat{y}))$

In classification problems, if we assume that the probability distribution $\hat{y}$ predicted by the model is obtained through the softmax function, then maximizing the log-likelihood function is equivalent to minimizing the cross-entropy loss.

Specifically, assume we have a neural network for a classification task, and the output layer uses the softmax activation function to convert the model output into category probabilities. For each sample $x_i$, its true label is $y_i$, and the probability predicted by the model is $\hat{y}_i$. Maximizing the log-likelihood function:

$\ell(\theta) = \sum_{i=1}^n \log P(y_i|x_i; \theta) = \sum_{i=1}^n \log \hat{y}_i$

is equivalent to minimizing the cross-entropy loss:

$H(y, \hat{y}) = - \sum_{i=1}^n \log \hat{y}_i$

Therefore, in this case, minimizing the cross-entropy loss when training the model is actually performing maximum likelihood estimation.

[W3]

As stated in line 203, we mentioned that different parameter freezing strategies are applied to the first $1 - L_F$ layers and the last $L_F - L_{F+U}$ layers. Here, $L_F$ is the number of frozen layers in the middle. The three tasks are trained separately.

[Q1]

We believe that in most cases, the selected $K$ words from the model will not contradict each other. This is because contradictory words are semantically different, meaning their distances in the embedding space are large. Thus, the score-matching function used in the HTPP module will rarely match contradictory words simultaneously.

[Q2]

As mentioned in line 225, the user embedding will not be used as input when performing the TUL task.

[Q3]

Our method requires full access to the implementation code and pre-trained parameters of an LLM to fine-tune some of the parameters. Since GPT-3.5 is not open-sourced and only limited functionalities can be accessed through the internet interface provided by OpenAI, GPT-3.5 cannot be used as the foundation for the proposed method.

[Limitations]

Most downstream tasks can be performed by attaching prediction modules (usually implemented with MLP) to the output latent vectors of LLM. For example, in our implementation, we use classification prediction modules for the LP and TUL tasks, and a regression prediction module for the TP task.

Regarding the choice of attachment position, an intuitive design is to attach it to the position corresponding to the most relevant information for the task. Experimental evaluation is also needed to determine the optimal position and implementation details.

Thank you for your thoughtful and constructive comments.

[W1&Q1]

In the paper, we ran each set of experiments 5 times and reported their mean values. Therefore, the reported results are not single accidental one. Below, we also report the variances of metrics for the proposed model and the best baseline on the Location Prediction task, along with the corresponding t-value measuring statistical significance.

The formula for calculating t-value is:

$t = \frac{\mu_{\text{Mobility-LLM}} - \mu_{\text{Best Baseline}}}{\sqrt{\frac{\delta_{\text{Best Baseline}}^2}{n}}}$

The standard deviation is that of the best baseline, with (n = 5). We use a significance level of $\alpha = 0.05$ and a two-tailed t-test with degrees of freedom $d_f = 4$. For this t-distribution, the critical t-value is approximately 2.776. If the calculated t-value is greater than 2.776, the difference is considered significant. As shown in the table, all the calculated t-values are well above the critical value of 2.776.

[W2&Q2]

Below, we provide an empirical performance and efficiency comparison between the proposed model and the optimal baseline for LP and TUL tasks.

Performance comparison for LP task:

Efficiency comparison for LP task:

Performance comparison for TUL task:

Efficiency comparison for TUL task:

We observe that our model increases memory usage due to the inclusion of LLM. However, since our model uses a pre-trained language model, its convergence speed is much faster than other baseline models.

On the high-quality Foursquare dataset, our model shows an improvement of over 21% compared to PLSPL, with the Acc@1 metric reaching up to 26%. Additionally, our model's advantage is more pronounced in the few-shot scenario. Therefore, the total training time is justified compared to traditional models.

In summary, we believe the increased cost is worthwhile for the performance improvement.

[W3&Q3]

LLM is designed to understand semantic information in natural language but not to interpret information in check-in sequences. Therefore, VIMN is proposed to reprogram the check-in sequence. This way, the output of VIMN provides semantic information aligned with natural language, allowing the LLM to process it.

[W4&Q4]

Sorry for causing misunderstandings. What we are trying to discuss in the Limitation section is that the sets of POIs in different datasets (which usually cover different regions) are unique. Therefore, if the proposed model is trained on one dataset, its learned information about the set of POIs is not easily transferable to another dataset. Different sets of POIs have different functionalities and usually have a different number of POIs, making many modules (such as embedding and predictor) technically untransferable in a zero-shot setting.

I thank the authors for providing additional experiment results. They should be included in the manuscript for completeness. However, I still think the performance gain is not substantial enough, especially considering the significantly larger model size. After reading other reviewers' comments, I lean towards a rejection now.