Reviewer’s Comment: What does "categorical location" mean? An example can be given to illustrate.

Response: We thank the reviewer for requesting clarification on "categorical location."

In our approach, each location is treated as a discrete category. In the YJMob100K dataset [1], the city is divided into a 200×200 grid, where each grid cell is a unique location category. Each location becomes a categorical ID that is then encoded using one-hot encoding. This allows the model to learn location-specific patterns efficiently.

In our revised version, we will add a clearer explanation in Section 3.2.

Reviewer’s Comment: The description of dividing into segments by day-of-week or hour-of-day is not clear enough there…

Response: We thank the reviewer for pointing out this in our segment description.

To clarify, $L$ represents the number of time steps (records) in each segment, not day-of-week or hour-of-day units. In the YJMob100K dataset, each record captures location every 30 minutes, so for our 1-day segments, $L$=48 (48 half-hour intervals = 24 hours).

The segmentation process works as follows: Each segment $s_i$ contains $L$=48 consecutive time steps representing one full day. For example, Day 1 segment contains records from 00:00-23 :30 (48 records), Day 2 segment contains records from 00:00-23:30 of the next day (48 records), and so on. $L$ is a tunable hyperparameter that users can adjust to match their desired segment length, similar to standard practice in time series forecasting.

The day-of-week and time-of-day mentioned in our temporal embeddings (line 117-119) are attributes within each timestamp, not segmentation units. Each of the 48 records within a segment has its own temporal embedding encoding both time-of-day (which hour) and day-of-week (Monday-Sunday).

In our revised version, we will clarify this to avoid any confusion about the segmentation granularity.

Reviewer’s Comment: In line 290, it is suggested that to compare the computing speed of the attention mechanism more fairly…

Response: We thank the reviewer for this valuable suggestion about comparing computational speed more fairly.

We have conducted additional experiments comparing RHYTHM with and without the LLM backbone to isolate the computational impact of our attention mechanism on the Kumamoto dataset. The table below shows the detailed timing comparison:

The results show that while incorporating the frozen LLM increases computation time by approximately 2.3×, it provides significant performance gains (+6.5% Acc@1). Notably, RHYTHM without LLM achieves comparable performance to PatchTST with similar computational cost, while outperforming iTransformer despite iTransformer’s higher computational overhead. The attention mechanism itself is highly efficient, demonstrating that our temporal tokenization design successfully reduces computational complexity as intended. The additional time from the frozen LLM is justified by the substantial accuracy improvements and is still more efficient than alternative LLM-based approaches.

In our revised version, we will include this detailed timing analysis to provide a more comprehensive view of the computational trade-offs in our design.

Reviewer’s Comment: There is a very confusing aspect of the innovation point… The third point of innovation is not very distinct.

Response: Thank you for the opportunity to clarify.

The novelty of our parameter-efficient fine-tuning lies in fully freezing the LLM backbone—including attention and FFN layers—while training only the hierarchical attention encoder and output head. Prior human mobility prediction models typically train from scratch, which is computationally expensive. Beyond this, our key contribution is enabling spatial-temporal data to interface with frozen LLMs. Since the latent space of such data differs fundamentally from textual inputs, we introduce hierarchical attention and prompt-guided semantic embeddings to bridge the modality gap. Unlike prior methods that use in-context learning [2, 3] or unfreeze the LLM [4], RHYTHM is the first to support structured spatial-temporal representation with frozen LLMs efficiently with less computational resources needed. Our novel temporal tokenization further reduces training and deployment costs by shortening sequence length while preserving temporal dependencies.

We will revise Section 3 to more clearly articulate these technical innovations and their distinctions from existing approaches.

Reviewer’s Comment: Generally, the prior feature extraction ability of LLM is utilized and should be verified in the generalization experiment, which can better reflect the significance of LLM.

Response: We thank the reviewer for this important question about verifying the LLM's generalization capabilities. To demonstrate that the pretrained LLM's feature extraction abilities specifically benefit mobility prediction, we conducted the following experiments:

These results clearly show that the pretrained LLM's prior knowledge is crucial - using a randomly initialized LLM performs worse than even our attention-only baseline. This verifies that the LLM's pretraining on diverse text enables it to extract meaningful spatio-temporal patterns, validating the significance of leveraging pretrained LLMs for mobility prediction.

Reviewer’s Comment: Ablation experiments can be conducted for more detailed verification.

Response: We thank the reviewer for suggesting more detailed ablation experiments. We have conducted comprehensive ablations on individual components of our spatio-temporal embeddings.

The results show that spatial information is most critical, while temporal embeddings provide complementary benefits. Interestingly, the model still achieves reasonable performance using only sequence order. However, our explicit spatio-temporal embeddings provide significant gains.

Reviewer’s Comment: The descriptions in some places are not clear enough.

Response: Thank you for the feedback. We will revise the manuscript to clarify the descriptions and improve overall readability in the relevant sections.

Reviewer’s Comment: None of the formulas in the article are numbered. Is this an operational error by the author? 2.There is?? In line 317.

Response: Thank you for pointing this out. The formulas are not numbered because they are not cross-referenced elsewhere in the paper - following standard practice, we only number equations that need to be referenced later in the text. Regarding the ?? in line 317, this is a LaTeX \ref typo. I apologize for this oversight and will ensure all reference placeholders are properly resolved in the revised version.

[1] Yabe, Takahiro, et al. "YJMob100K: City-scale and longitudinal dataset of anonymized human mobility trajectories." Scientific Data 11.1 (2024): 397.

[2] Wang, Xinglei, et al. "Where would i go next? large language models as human mobility predictors." arXiv 2023.

[3] Beneduce, Ciro, et al. "Large language models are zero-shot next location predictors." IEEE Access 2025.

[4] Gong, Letian, et al. "Mobility-llm: Learning visiting intentions and travel preference from human mobility data with large language models." NeurIPS 2024.

Reviewer's Comment: While the reproducibility aspect is strong, the paper does not provide explicit information about the computational…

Response: We thank the reviewer for raising this important point for reproducibility. We provide comprehensive computational resource requirements below. All experiments are run on a single NVIDIA A100 GPU (40GB) with batch size 64.

Further reproducibility details of inference, pre-computation, and deployment are discussed in the following responses.

Reviewer’s Comment: You said "This design captures fine-grained spatio-temporal dynamics, deep semantic context, and leverages LLM reasoning…

Response: We thank the reviewer for requesting concrete deployment metrics to support our efficiency claims. We have conducted comprehensive benchmarking on both GPU and CPU to demonstrate RHYTHM's suitability for resource-constrained environments.

This benchmarking was performed on NVIDIA A100 (GPU) and Intel Xeon Gold 6338 (CPU) with 100 samples. RHYTHM maintains a peak GPU memory (GRAM) footprint of 8.2 GB, significantly lower than TimeLLM's 22.4 GB, enabling deployment on more modest hardware. All models show expected slowdowns on CPU, but RHYTHM's 39.5s latency remains practical for batch processing scenarios. We will expand our discussion on practical deployment considerations in resource-constrained environments in the revised manuscript.

Reviewer’s Comment: The choice of daily segments (L = 48 time slots) for temporal tokenization is justified by general human routines…

Response: We thank the reviewer for raising this important question about the sensitivity of our temporal tokenization to segmentation granularity. We have conducted a comprehensive sensitivity analysis varying the segment length to understand its impact on performance and computational cost.

Our sensitivity analysis reveals that RHYTHM's performance peaks at $L$=48 (daily segments), with a 4.5% improvement over the finest granularity ($L$=1) and 2.7% over coarser segmentation ($L$=96). The computational cost scales dramatically with smaller segments due to quadratic attention complexity - $L$=1 requires 6.9× more computation than $L$=48.

This empirical validation confirms that daily segmentation is not just theoretically motivated by human routines but also optimal in practice. The performance degradation at other granularities demonstrates that our temporal tokenization effectively captures the natural periodicity of human mobility patterns.

In our revised version, we will include this sensitivity analysis in Section 4.2, providing deeper insights into how segmentation granularity affects the balance between capturing fine-grained patterns and computational efficiency.

Reviewer’s Comment: The paper enriches segment tokens with pre-computed LLM prompt embeddings for both historical trajectories and future timestamps…

Response: We thank the reviewer for highlighting the need to quantify the offline computation and storage costs of our semantic embeddings. We have conducted detailed measurements across all three datasets.

These measurements were performed on a single NVIDIA A100 GPU (40GB memory) with batch size 128. The memory overhead is 31.24 GB. The storage requirements include both trajectory and task embeddings.

The generation time scales linearly with user count, and storage remains manageable even for 22k users. This one-time preprocessing eliminates redundant LLM inference during training, reducing epoch time from hours to minutes while enabling deployment on resource-constrained hardware without LLM infrastructure. We will add a dedicated section on these costs in the revised manuscript.

Reviewer’s Comment: While Appendix B discusses technical limitations—reliance on pretrained LLM quality …

Response: We thank the reviewer for encouraging us to provide more concrete deployment guidance.

We summarize the training setup and performance of RHYTHM across different devices and sequence lengths in the tables below.

As demonstrated in our previous response to question 1, RHYTHM can run on CPU-only devices (39.5s inference), making it viable for edge deployment where real-time constraints are relaxed. For resource-constrained scenarios like a single RTX 2080, we have verified that using batch size 8 with up to 75 days of history is feasible to fit within 8GB memory constraints. Due to our temporal tokenization strategy, RHYTHM handles long sequences with less memory and faster speed than previous methods.

Our frozen LLM design and temporal tokenization strategy makes RHYTHM practical across diverse deployment scenarios from edge devices to cloud servers. We will expand Appendix B with deployment guidelines in our revised version.

Reviewer’s Comment: In Section 4.2, the text refers to an ablation result “as shown in ??,”...

Response: Thanks for catching this! This is a LaTeX \ref typo that has already been fixed in the manuscript. All "??" placeholders will be replaced with the correct Table/Figure numbers.

Thank you again for your thoughtful questions and comments.

We would appreciate it if you could let us know whether our responses have sufficiently addressed your main concerns, or if there are any points that would benefit from further clarification. We welcome any additional feedback and would be happy to elaborate as needed.

Reviewer’s Comment: Novelty is limited…

Response: We thank the reviewer and respectfully clarify our novelty:

• First to bridge frozen LLMs with structured spatio-temporal data: RHYTHM is the first to combine hierarchical temporal abstraction with semantic prompt injection into a frozen LLM. Unlike in-context learning [1] or fine-tuning-based methods [2], we freeze the entire LLM—including attention and FFN layers—and only train a lightweight hierarchical encoder and projection head. This design is parameter-efficient (12.37% trainable parameters) and scalable to large models.
• First to demonstrate LLM scaling laws in the mobility domain: We conduct a comprehensive analysis (Table 4, Fig. 4) across multiple LLM backbones (LLaMA, DeepSeek, Gemma, …) to quantify trade-offs between model size, training time, and prediction accuracy—a first in human mobility modeling.
• Efficient hierarchical architecture for multi-scale mobility dynamics: Compared to prior work, existing human mobility foundation models either rely on in-context learning [1] or require fine-tuning within the LLM backbone [2]. However, fine-tuning FFN layers incurs substantial computational cost. In contrast, we avoid modifying the LLM backbone. Instead, we enable spatio-temporal and semantic fusion by training only the hierarchical attention encoder and generation head. This approach significantly reduces resource demands while maintaining strong generative capacity over spatial-temporal data, which lies in a fundamentally different latent space than text.
• Hierarchical temporal tokenization for multi-scale mobility modeling: While prior work (e.g., PMT, ST-MoE-BERT) employs standard attention over long sequences, RHYTHM introduces temporal tokenization that encodes daily mobility patterns as discrete tokens. These tokens are processed through explicit hierarchical attention to capture both short-term (daily) and long-term (weekly) periodic dependencies. Crucially, our design eliminates the need for full-length spatio-temporal sequences, enabling the model to efficiently and accurately reason over long mobility histories without incurring the quadratic cost of full-sequence attention. This leads to both reduced training overhead and improved scalability.
• Empirical strength across real-world datasets: RHYTHM demonstrates strong real-world performance, achieving a 2.4% improvement in overall prediction accuracy, a 5.0% gain on weekends, and a 24.6% reduction in training time over prior state-of-the-art baselines (Table 1, Fig. 3, Fig. 5). These results are consistent across three urban mobility datasets. Taken together, RHYTHM provides a novel, efficient, and scalable framework for mobility prediction. It is not a repackaging of prior ideas but a careful integration that enables structured non-textual inputs to leverage the reasoning power of LLMs—without requiring costly full-model fine tuning. In our revised version, we will clarify our writing to better highlight our novel contributions.

Reviewer’s Comment: The models used in the performance experiment…

Response: We thank the reviewer for this important clarification about model consistency and presentation. We apologize for the confusion in our presentation, and we will revise our tables accordingly. Importantly, we want to emphasize that even with LLaMA-1B model, RHYTHM achieves 1.8% improvement over the best baseline, demonstrating strong performance without requiring larger models.

Reviewer’s Comment: The efficiency experiment design is too simple…

Response: We thank the reviewer for highlighting the need for more detailed efficiency experiments. We acknowledge this oversight and have conducted comprehensive efficiency analysis for each component.

Here are the detailed efficiency measurements on the Kumamoto dataset (single A100 GPU):

These results demonstrate that each design choice contributes to efficiency. To clarify the metrics mentioned in our contributions: The 12.37% refers to trainable parameters. The 24.6% reduction in training time is calculated by comparing RHYTHM with mobility prediction baselines. In our revised version, we will add this clear explanation in the experiment section.

Reviewer’s Comment: In the ablation experiment, each module…

Response: We thank the reviewer for this important feedback about our ablation study. We conducted a more comprehensive analysis to clarify each component's contribution.

The results reveal that while individual components show modest improvements, removing the LLM backbone causes a significant 6.2% performance drop, demonstrating that our approach's strength comes from the synergistic integration of LLM reasoning with mobility-specific components. Regarding temporal tokenization, while segmentation techniques exist in other domains, our specific application to mobility prediction is novel - as illustrated in Figure 1, we segment trajectories into semantically meaningful daily patterns rather than arbitrary windows, enabling the model to capture cyclical human behaviors. This contributes 5.4% improvement when integrated with our LLM framework.

In our revised version, we will update the ablation section with clearer explanations of how each component contributes to the overall system performance.

Reviewer’s Comment: From the method section, the number of segments N…

Response: We thank the reviewer for identifying this important parameter that deserves more detailed analysis. We have conducted comprehensive experiments with different segment lengths to understand their impact on both performance and efficiency.

The results reveal an interesting trade-off: $L$=48 (daily segments) achieves optimal performance, aligning with natural mobility rhythms. Smaller segments ($L$=1) suffer from excessive fragmentation and computational overhead, while larger segments ($L$=96) lose important daily pattern boundaries.

Our choice of $L$=48 captures complete daily cycles while maintaining computational efficiency. This validates that temporal tokenization at semantically meaningful boundaries is more effective than arbitrary segmentation.

In our revised version, we will add this ablation study and include a detailed discussion on how segment length affects the model's ability to capture mobility patterns at different temporal scales.

Reviewer’s Comment: The results in Daily and Weekly…

Response: Thank you for raising this important point about the seemingly contradictory performance patterns. This is a crucial finding that reveals the fundamental strength of our approach. This contradiction with our claims about modeling periodicity actually reveals a deeper insight about our approach.

RHYTHM's hierarchical attention mechanism captures periodicity not just as repetitive patterns but as contextual decision points. During highly regular periods, mobility is largely deterministic - people follow fixed routines with minimal variation. Traditional models like LSTM and PMT, which rely on simple pattern matching, excel here because the prediction task reduces to memorizing frequent transitions.

However, during transitional periods (evening peaks) and flexible periods (weekends), mobility involves complex decision-making influenced by multiple factors. Here, RHYTHM's advantage emerges:

• The LLM backbone provides reasoning capabilities to handle non-routine decisions
• Hierarchical attention captures both local context (what happened today) and global patterns (typical weekend behavior)
• Semantic prompts encode rich contextual information beyond simple location sequences

Hierarchical attention is optimized for capturing global temporal dependencies across days and weeks, which may actually dilute its focus on simple repetitive patterns within regular hours. RHYTHM considers broader context while powerful for complex decisions, and introduces unnecessary complexity for deterministic periods.

This explains the 5.0% improvement on weekends - RHYTHM excels when prediction requires understanding context and variability rather than just memorizing fixed patterns. During regular weekday hours, the simpler baseline models are sufficient because the task is essentially deterministic.

In our revised version, we will clarify that our "periodicity modeling" refers to capturing periodic variations in behavior complexity rather than just repetitive patterns, making RHYTHM particularly valuable for real-world applications where handling irregular periods is crucial for overall system reliability.

[1] Wang, Xinglei, et al. "Where would i go next? large language models as human mobility predictors." arXiv 2023.

[2] Gong, Letian, et al. "Mobility-llm: Learning visiting intentions and travel preference from human mobility data with large language models." NeurIPS 2024.

Thank you again for your thoughtful questions and comments.

We would appreciate it if you could let us know whether our responses have sufficiently addressed your main concerns, or if there are any points that would benefit from further clarification. We welcome any additional feedback and would be happy to elaborate as needed.

As a gentle follow-up to our earlier comment, we wanted to kindly remind you that we have already provided comprehensive responses and additional experiments in our rebuttal, addressing all raised concerns.

With the discussion period nearing its end, we would greatly appreciate any remaining feedback from you, to ensure all points are fully addressed

Reviewer’s Comment: The frozen LLM backbone may not adapt to domain-specific nuances …

Response: We thank the reviewer for this insightful comment. We address both concerns below:

• Frozen LLM Backbone and Domain-Specific Nuances: We appreciate the reviewer's perspective and would like to clarify that the frozen backbone is a deliberate design choice that offers several advantages for domain adaptation:
  • While the LLM backbone remains frozen, domain-specific adaptation occurs through lightweight learnable components: spatio-temporal embeddings capture location-specific patterns, temporal tokenization models region-specific daily/weekly rhythms, and learnable projection layers map domain-specific features into the LLM's representation space. The frozen LLM serves as a powerful representation extractor that processes these domain-adapted features through its deep reasoning capabilities, which has been explored in [1, 2].
  • Our experiments demonstrate successful adaptation across three distinct cities. The model achieves consistent performance improvements (2.4-2.7% over best baselines), despite their different urban characteristics and transit patterns. This validates that our architecture effectively captures regional behavioral differences without modifying the LLM backbone.
  • From a theoretical perspective, as noted in Section 3.4, frozen LLMs provide guaranteed convergence properties and uniform feature distribution, ensuring stable learning of domain-specific patterns through the trainable components.
• Static Semantic Prompts: Our semantic prompts are contextually rich and incorporate dynamic information. Each prompt is specifically generated based on user trajectory segments, temporal context (day-of-week, time patterns), segment-specific transitions and stay patterns, and task-specific prediction requirements.
  • By generating these embeddings offline before training, we eliminate the need for expensive LLM forward passes during the training process. For the Kumamoto dataset, the one-time prompt generation and processing takes approximately 3 hours on a single A100 with 40G memory.
  • We acknowledge the reviewer's suggestion about dynamic prompt learning. In our revised version, we will add a discussion on prompt tuning techniques that could potentially enhance performance by allowing prompts to adapt during training.

Reviewer’s Comment: While temporal patterns are captured hierarchically, spatial information is encoded via simple embeddings without incorporating topology…

Response: We thank the reviewer for this valuable observation about spatial modeling and spatio-temporal relationships.

Our current approach uses categorical location embeddings with coordinate projections rather than incorporating spatial topology or explicit relational structures. We acknowledge that including topological information and explicit spatio-temporal relationships could provide additional spatial understanding and potentially enhance performance in connectivity-aware tasks.

However, our current design prioritizes computational efficiency and scalability, which motivated our choice of joint spatio-temporal embeddings. This design decision allows us to maintain training efficiency (24.6% faster) while still achieving state-of-the-art performance (2.4% improvement over baselines), suggesting that the LLM's reasoning capabilities partially compensate for the simplified spatial representation.

We agree that explicit modeling of spatial relationships is an important direction that we will explore in future work.

Reviewer’s Comment: Evaluation is confined to the Japanese-only YJMob100K dataset; results on other continents or longer temporal spans are important yet missing (longer than 1 day window).

Response: We thank the reviewer for raising this important point about dataset diversity and temporal evaluation scope. Mobility datasets are inherently limited due to privacy concerns and collection difficulties. YJMob100K [3] is one of the few publicly available large-scale mobility datasets with 75 days of data and sufficient user records, ensuring reproducibility of our results.

To address the reviewer's concern, we have conducted additional experiments on a Boston dataset with the same configuration (10k users). The results show consistent performance improvements, demonstrating that RHYTHM generalizes beyond Japanese cities.

Regarding longer temporal spans, we conducted experiments with 3-day prediction horizons on the Kumamoto dataset. The results indicate that RHYTHM still outperforms most of the baselines in this configuration.

Reviewer’s Comment: Predictions are made in a single shot (non-autoregressive), without iterative feedback, performance on rolling horizons remains untested.

Response: We thank the reviewer for highlighting this important aspect of our prediction approach. To address this concern, we have conducted additional experiments comparing autoregressive and non-autoregressive prediction strategies. Table below shows the performance and computational speed comparison on rolling horizons.

The results demonstrate that our non-autoregressive approach achieves comparable performance while being 41× faster than the autoregressive variant. This significant computational advantage makes RHYTHM particularly suitable for real-time deployment scenarios. In our revised version, we will add a detailed discussion on why our temporal tokenization with hierarchical attention provides sufficient context for accurate multi-step predictions without requiring expensive iterative feedback mechanisms.

[1] Jin, Ming, et al. "Time-llm: Time series forecasting by reprogramming large language models."ICLR 2024.

[2] Liu, Yong, et al. "Autotimes: Autoregressive time series forecasters via large language models." NeurIPS 2024.

[3] Yabe, Takahiro, et al. "YJMob100K: City-scale and longitudinal dataset of anonymized human mobility trajectories." Scientific Data 2024.

Thank you again for your thoughtful questions and comments.

We would appreciate it if you could let us know whether our responses have sufficiently addressed your main concerns, or if there are any points that would benefit from further clarification. We welcome any additional feedback and would be happy to elaborate as needed.