L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Models

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1: The evaluation is limited to post-training scenarios. The potential impact of L-MTP on pre-training LLMs remains unexplored, leaving an important aspect of its applicability unverified.

Response

Thanks! We acknowledge that pre-training would bring more impressive results, according to our potential analysis in Section 5. The effectiveness of pre-training for MTP is well-verified [1]. However, due to resource constraints, we could not pre-train LLMs with L-MTP. We believe future pre-training efforts can leverage L-MTP’s potential, which motivated our inclusion of the potential analysis.

W2: The comparison between L-MTP(k=2,n=4) and MTP(n=7) is missing, where both methods align in maximum prediction position. It would be valuable to examine whether L-MTP maintains its acceleration advantage over MTP(n=7) under the same candidate length.

Response

Thanks! We conduct the experiments when setting $n=7$ for MTP. A longer horizon indeed benefits the performance, achieving an average performance of 54.49 compared to 52.79 for MTP ($n=4$).

However, MTP ($n=7$) nearly doubles the heads (new parameters) compared to L-MTP ($k=2,n=4$ and $k=3,n=3$). Despite this, L-MTP delivers comparable or superior performance (e.g., 55.18 for $k=3,n=3$) with greater efficiency, particularly on GSM8K and HumanEval+. L-MTP is much more efficient in achieving a comparable yet sometimes better performance. The inference acceleration of L-MTP, enabled by the leap strategy and look-backward decoding, allows it to outperform MTP with the same resources (4 heads). The theoretical analysis in Section 4 supports this, establishing MTP ($n=7$) as the upper bound for L-MTP ($k=2,n=4$) speedup. We will include this analysis in the revised paper.

W3: Experiments are only conducted on the (k=2,n=4) configuration. What is the performance of other configurations, for example, (k=3, n=3)?

Response

Thanks! Your suggestion is reasonable to further demonstrate the application of L-MTP. We conducted this experiment as shown in W2 and the corresponding response. Overall, L-MTP ($k=3, n=3$) performs better than others, especially the improvements on GSM8K and HumanEval. Exploring more leap patterns would be promising. We also discuss a potential leap pattern via entropy estimation in Section 7 to showcase a broader application of L-MTP. We will include these results in our revised paper.

Q1: The work in [1] would be a valuable addition to the references, as it provides complementary insights on multi-token prediction for accelerating LLM training.

Response

Thanks for your share! This is a very interesting work, which predicts patches of tokens for LLMs by aggregating adjacent tokens with higher information density. This aligns closely with our discussion in Section 7 on extending L-MTP with flexible leaps, where we propose skipping low-entropy regions to focus on high-entropy ones. Combining [1] with L-MTP could create significant synergy. We will incorporate a discussion of this paper and explore further possibilities in our revised manuscript.

[1] Shao et al. Beyond next token prediction: Patch-level training for large language models. ICLR 2025.

We authors sincerely thank you for your professional review attitude and comments. If you have other questions, we are happy to address them to polish this work.

Thanks for your kind reminder. Due to the averaging computation, we mistakenly reported the average performance of L-MTP ($k=3,n=3$) and MTP ($n=7$) in the original submission. We carefully re-evaluate their average performance and check other results to ensure a correct report as follows:

We can see that directly increasing the horizon of MTP does not improve the performance overall. But we can still observe some improvement on HumanEval. The interesting thing is that when we decrease the number of heads to 3, L-MTP ($k=3,n=3$), it can achieve a better performance than MTP ($n=7$). Our theory on the prediction analysis at different positions can answer this (Section 4). Distant tokens would lead to noise, while our leap can catch future tokens, but also leap some. An accumulated noise would be smaller than MTP. A smaller number of heads ($n=3$) while with the leaping strategy can achieve a comparable performance (52.79, MTP with $n=4$) or outperform MTP (51.82, $n=7$).

We apologize for the inconvenience and thank you for your positive feedback on our paper, and for your reminders during the rebuttal process. We will incorporate these results and discussion into the final version. If you have any further questions or concerns, please feel free to contact us.

Here are the responses to the relevant questions:

W2: The paper missing the ablation of the horizon (MTP with n=7), therefore it is not clear whether the improvement comes simply from having a longer horizon or something else entirely. Without this, researchers don't gain a better understanding of MTP and practitioners don't have clear evidence to support L-MTP over MTP with n=7.

Response

Thanks! We conduct the experiments when setting $n=7$ for MTP. A longer horizon can benefit the performance for some benchmarks, especially for HumanEval, the code tasks (68.29 vs. 65.85), while showing slightly worse performance on the overall average (51.82 vs. 52.79). The results are shown above. MTP ($n=7$) nearly doubles the heads (new parameters) compared to L-MTP ($k=2,n=4$ and $k=3,n=3$). Despite this, L-MTP delivers comparable (52.35 for $k=3,n=3$) or superior performance (e.g., 53.75 for $k=2,n=4$) with greater efficiency, particularly on GSM8K and HumanEval+. L-MTP is much more efficient in achieving a comparable yet sometimes better performance. The inference acceleration of L-MTP, enabled by the leap strategy and look-backward decoding, allows it to outperform MTP with the same resources (4 heads). The theoretical analysis in Section 4 supports this, establishing MTP ($n=7$) as the upper bound for L-MTP ($k=2,n=4$) speedup. We will include this analysis in the revised paper.

W3: Experiments are only conducted on the (k=2,n=4) configuration. What is the performance of other configurations, for example, (k=3, n=3)?

Response

Thanks! Your suggestion is reasonable to further demonstrate the application of L-MTP. We conducted this experiment as shown in W2 and the corresponding response. Overall, L-MTP ($k=3, n=3$) does not outperform L-MTP ($k=2,n=4$), but shwocases improvements on GSM8K and HumanEval. That indicates the leaping strategy can enhance certain capabilities for different benchmarks. Different leaping settings provide more flexibility for different tasks. Exploring more leap patterns would be promising. We also discuss a potential leap pattern via entropy estimation in Section 7 to showcase a broader application of L-MTP. We will include these results in our revised paper.

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1: While the paper is clearly written and the results show some improvement over established baselines, the contribution appears to be incremental. The core idea is relatively simple, and the source of the observed improvement is neither thoroughly ablated nor adequately discussed, which limits the scientific insight provided.

Response

Thank you for raising this concern, and we’re happy to clarify our contribution! MTP, introduced in recent works [1,2], predicts the next $n$ adjacent tokens simultaneously, inspiring applications like speech language models and inference acceleration. The insight, not every token is equally important, is validated in LLMs [3] and aligned with human thinking. This inspires us, yet disappointingly, no research focuses on new prediction patterns, especially the non-adjacent token prediction, a leap paradigm against the recent adjacent prediction of MTP. L-MTP introduces this leap paradigm with look backward decoding, achieving broader and faster predictions than MTP using the same resources (training data and model architecture). We analyze this comprehensively through prediction principles (Section 4) and empirical verification (Section 5). Our theoretical analysis links L-MTP to probabilistic attenuation, showing that lower attenuation yields greater speedups. The empirical analysis demonstrates consistent improvements and provides in-depth insights (e.g., Section 5, Potential Analysis), highlighting L-MTP’s strong promise through analyses of myopic effects and data scales.

[1] Gloeckle, Fabian, et al. "Better & faster large language models via multi-token prediction." ICML 2024.
[2] Liu, Aixin, et al. "Deepseek-v3 technical report." arXiv preprint arXiv:2412.19437 (2024).
[3] Lin, Zhenghao, et al. "Rho-1: Not all tokens are what you need." NeurIPS 2024.

W2: The paper missing the ablation of the horizon (MTP with n=7), therefore it is not clear whether the improvement comes simply from having a longer horizon or something else entirely. Without this, researchers don't gain a better understanding of MTP and practitioners don't have clear evidence to support L-MTP over MTP with n=7.

Response

Thank you for your interest in MTP with $n=7$, a concern also raised by Reviewer SKtJ. We did not evaluate MTP with $n=7$, as our study ensures a fair comparison by using the same resources (both MTP and L-MTP have 4 heads). Setting $n=7$ for MTP would nearly double the new parameters compared to L-MTP, making it an unfair comparison rather than a proper ablation study. However, we are open to conducting this experiment to provide further insights. Due to time and resource constraints, we cannot test this across all LLM scales and types. Below are the results:

Increasing MTP's heads ($n=7$) improves performance for its longer horizon (54.49 vs. 52.79). However, L-MTP achieves comparable results with nearly half the new parameters (4 vs. 7 heads). Additionally, the new leap pattern ($k=3,n=3$) yields higher performance (55.18 vs. 54.49), particularly on GSM8K and HumanEval+. We will incorporate this analysis into our revised paper.

W3: The paper would be much more impactful had it figured out the source of the improvement (both in performance and inference speedup).

Response

Thanks! We agree with you that the source of performance improvement and inference speedup should be clarified. That is the reason why we provide a thorough theoretical analysis and extensive empirical results. L-MTP achieves a broader prediction horizon with the same resources as MTP by skipping intermediate tokens. Beyond intuitive motivation and sophisticated implementation, we provide an in-depth analysis.

Inference Aspect: We formally link token prediction to attenuation, supporting L-MTP’s inference speedup. While a longer horizon increases uncertainty, L-MTP’s look-backward decoding utilizes prior predictions, thus yielding higher expected prediction lengths than MTP. We also introduce a decoding variant, F-MTP, for comparison, which uses look-forward decoding to compensate for leaped tokens but requires an additional inference step, increasing inference time (see Figure 6). The combination of leaping prediction and look-backward decoding drives L-MTP’s inference efficiency.

Performance Aspect: Inspired by the insight that not every token matters [3] and human thinking, L-MTP aligns with recent work on reasoning step compression and abstraction [4, 5]. The longer horizon is key to performance gains, as seen in meaningful units like code blocks versus single tokens (e.g., {). Experiments with MTP at $n=7$ further validate this. L-MTP’s leap design enhances performance by using two tokens (when $k=2$) to predict $m$ tokens, creating a flexible mapping that correlates both input and output tokens (2-to-m tokens), unlike MTP’s single-token-to-m-token approach. Increasing $k$ while keeping the number of output tokens can further boost performance, as observed in the results of L-MTP at $k=3,n=3$.

[4] Chen, Zhipeng, et al. "Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment." EMNLP 2024.
[5] Xia, Heming, et al. "Tokenskip: Controllable chain-of-thought compression in llms." arXiv preprint arXiv:2502.12067 (2025).

W4: Also, there could be many variants of L-MTP with different pacing, variable pacing, longer/shorter horizons etc. I feel the paper had the opportunity to explore a larger idea space, but did not do so.

Response

We explore L-MTP with different horizons to identify an effective leaping and horizon configuration for evaluation. Our theoretical analysis indicates that longer horizons introduce noise due to increased uncertainty in predicting distant tokens, while shorter horizons (e.g., $n=2$) may not fully leverage multi-token prediction. We select $k=2, n=4$ as a practical balance, aligning with prior MTP methods for a fair and effective comparison. We appreciate your suggestions and conduct additional experiments with $k=3, n=3$, with results discussed earlier. Exploration of more flexible leaping patterns is planned as future work, building on L-MTP as shown in Section 7.

Q1: The source of the improvement is unclear (applied both the performance improvement and inference speedup): The most obvious missing baseline is MTP with n=7. Is the improvement coming from the longer horizon or is it something else? If it is the longer horizon, then what is the advantage of LMPT over MTP with n=7? If it is something else, then it is an interesting scientific question of what that may be.

Response

Thank you for your question! We address the source of L-MTP’s improvement in W3 and the corresponding response, supported by additional empirical evidence in W2. Our theoretical analysis and empirical results demonstrate L-MTP’s superiority in both performance and inference. The additional potential analysis further offers insights on MTP and the significant promise of L-MTP. Results from L-MTP ($n=2, k=4$ and $n=3, k=3$) and MTP ($n=4$ and $n=7$) show that performance gains can benefit from a longer horizon.

We sincerely appreciate your review and hope our response, bolstered by additional empirical results, aids in re-evaluating our work.

Thanks for the kind reminder of the reviewer SKtJ. Due to the averaging computation, we mistakenly reported the average performance of L-MTP ($k=3,n=3$) and MTP ($n=7$) in the original submission. We carefully re-evaluate their average performance and check other results to ensure a correct report as follows:

We can see that directly increasing the horizon of MTP does not improve the performance overall. But we can still observe some improvement on HumanEval. The interesting thing is that when we decrease the number of heads to 3, L-MTP ($k=3,n=3$), it can achieve a better performance than MTP ($n=7$). Our theory on the prediction analysis at different positions can answer this (Section 4). Distant tokens would lead to noise, while our leap can catch future tokens, but also leap some. An accumulated noise would be smaller than MTP. A smaller number of heads ($n=3$) while with the leaping strategy can achieve a comparable performance (52.79, MTP with $n=4$) or outperform MTP (51.82, $n=7$).

We are trying to provide a more comprehensive analysis of the performance of L-MTP and MTP during this time-limited rebuttal process. We apologize for the inconvenience and hope the following revised response can address your concerns. We will incorporate these results and discussion into the final version. If you have any further questions or concerns, please feel free to contact us.

Here are the responses to the relevant questions:

W2: The paper missing the ablation of the horizon (MTP with n=7), therefore it is not clear whether the improvement comes simply from having a longer horizon or something else entirely. Without this, researchers don't gain a better understanding of MTP and practitioners don't have clear evidence to support L-MTP over MTP with n=7.

Response

Thank you for your interest in MTP with $n=7$, a concern also raised by Reviewer SKtJ. We did not evaluate MTP with $n=7$, as our study ensures a fair comparison by using the same resources (both MTP and L-MTP have 4 heads). Setting $n=7$ for MTP would nearly double the new parameters compared to L-MTP, making it an unfair comparison rather than a proper ablation study. However, we are open to conducting this experiment to provide further insights. Due to time and resource constraints, we cannot test this across all LLM scales and types. Below are the results:

Increasing MTP's heads ($n=7$) cannot directly improve the overall performance. We attribute this to the noise introduced by distant tokens (see our theoretical analysis in Section 4). However, L-MTP achieves superior results with nearly half the new parameters (4 vs. 7 heads). Additionally, the new leap pattern ($k=3,n=3$) yields comparable performance with a decreased number of heads (3 vs. 4 heads for MTP), and showcases higher performance, particularly on GSM8K and HumanEval+, the benchmarks that evaluate the reasoning capability of LLMs. We will incorporate this analysis into our revised paper.

W3: The paper would be much more impactful had it figured out the source of the improvement (both in performance and inference speedup).

Response

Thanks! We agree with you that the source of performance improvement and inference speedup should be clarified. That is the reason why we provide a thorough theoretical analysis and extensive empirical results. L-MTP achieves a broader prediction horizon with the same resources as MTP by skipping intermediate tokens. Beyond intuitive motivation and sophisticated implementation, we provide an in-depth analysis.

Inference Aspect: We formally link token prediction to attenuation, supporting L-MTP’s inference speedup. While a longer horizon increases uncertainty, L-MTP’s look-backward decoding utilizes prior predictions, thus yielding higher expected prediction lengths than MTP. We also introduce a decoding variant, F-MTP, for comparison, which uses look-forward decoding to compensate for leaped tokens but requires an additional inference step, increasing inference time (see Figure 6). The combination of leaping prediction and look-backward decoding drives L-MTP’s inference efficiency.

Performance Aspect: Inspired by the insight that not every token matters [3] and human reasoning, L-MTP aligns with recent work on reasoning step compression and abstraction [4, 5]. The longer horizon is key to performance gains, as seen in meaningful units like code blocks versus single tokens (e.g., {). However, directly increasing the horizon does not benefit the overall performance. Based on our analysis of the prediction accuracy at different positions, we attribute it to the noise introduced by the tokens from distant positions. While L-MTP can reach the future tokens while also leaping some to reduce the accumulated noise. Additionally, L-MTP’s leap design enhances performance by using two tokens (when $k=2$) to predict $m$ tokens, creating a flexible mapping that correlates both input and output tokens, unlike MTP’s single-token-to-n-token approach.

[4] Chen, Zhipeng, et al. "Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment." EMNLP 2024.
[5] Xia, Heming, et al. "Tokenskip: Controllable chain-of-thought compression in llms." arXiv preprint arXiv:2502.12067 (2025).

We sincerely appreciate your professional review attitude and comments. We also notice that you have updated your review. We have carefully addressed your concerns. If you have any further questions or concerns, please feel free to contact us.

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1: Limited improvement: While the proposed leap-based training objective is conceptually interesting, the empirical gains over standard baselines are relatively modest.

Response

Thank you for your recognition of the novelty of our work! L-MTP focuses on both the performance boosting and inference acceleration [1], which is more challenging than one goal in either performance [2] or decoding acceleration [3, 4]. However, L-MTP achieves both goals, particularly for the inference speed-up. For a comprehensive performance evaluation, we conducted extensive experiments across various scales, LLM types, and tasks, observing consistent performance improvements. We also noted limited performance in certain cases, discussed in Section 5.2, attributing these to data quality. Selecting higher quality data would boost the performance with a branch of work that focuses on how to synthesize the data or collect new data to provide new knowledge for well-trained LLMs. However, in our work, we do not involve the data selection as our research topic. Additionally, our potential analysis in Section 5.2, Line 245, demonstrates that with more data or training resources (e.g., pre-training), L-MTP could achieve even greater performance gains. These experiments and analyses highlight the superiority and significant potential of L-MTP.

[1] Gloeckle, Fabian, et al. "Better & faster large language models via multi-token prediction." ICML 2024.
[2] Chen, Michael K., et al. "Improving large language models with concept-aware fine-tuning." arXiv preprint arXiv:2506.07833 (2025).
[3] Cai, Tianle, et al. "Medusa: Simple LLM inference acceleration framework with multiple decoding Heads." ICML 2024.
[4] Ankner, Zachary, et al. "Hydra: Sequentially-dependent draft heads for medusa decoding." COLM 2024.

W2: Low absolute performance: On challenging benchmarks such as MATH-500 and GSM8K, the model’s absolute performance remains low, raising doubts about whether L-MTP meaningfully improves LLM capabilities in practice.

Response

Thanks for your sincere review! The absolute performance of L-MTP heavily depends on the pretrained LLMs, with stronger models yielding better results. For example, Qwen outperforms Llama and Gemma on MATH500 and GSM8K. Our experiments across various LLM scales and types confirm that larger models achieve higher absolute performance. Thus, integrating L-MTP with more advanced or larger-scale LLMs, such as DeepSeek-R1-70B/671B or Llama-3.1-405B, would further enhance performance.

L1: L-MTP shows limited improvement over baselines, and its low scores on challenging benchmarks raise doubts about its effectiveness in enhancing LLM reasoning capabilities.

Response

Thanks! Achiving both significant performance improvement and inference acceleration is highly challenging, as discussed in W1 and its related response. Despite this, L-MTP demonstrates overall improvements, and some notable gains in reasoning-focused benchmarks like Math500 and GSM8K. Specifically, Qwen trained with L-MTP achieves a higher performance of 28.20 (vs. MTP’s 25.40) and Gemma achieves 17.20 (vs. MTP’s 9.20). A stronger base model can indeed lead to a higher absolute performance as discussed in W2. Due to the limited resources, we do not involve more powerful LLMs as our starting point for training. We also discuss this in Appendix G (Limitations). We value the practical application and broad impact of L-MTP, which motivates us to conduct an in-depth analysis for its potential (see Section 5 Potential Analysis). We investigate the myopia of current models with increased parameter size and the prediction accuracy in terms of increased data scales. It suggests that the significant potential of L-MTP with more tunable parameters (even from pre-training) or more data.

We authors sincerely thank you for your professional review attitude and comments. If you have other questions, we are happy to address them to polish this work.

Thank you for your response! I understand the limitations imposed by resource constraints, and I appreciate your efforts in conducting the current experiments. While I see the potential of the proposed approach, I feel that its practical applicability in real-world LLM applications is not yet fully demonstrated. For this reason, I will maintain my original score. That said, I look forward to seeing how this line of work evolves in the future.

Thanks for your professional and careful review. We respond to your concerns or questions as follows.

W1: The paper could be strengthened by a brief discussion of alternative designs for the leap mechanism, for instance, exploring leaps with non-uniform intervals instead of the current fixed stride.

Response

Thank you for your suggestion! We outline a more flexible leap pattern for L-MTP in Section 7, Line 297, as our future exploration. We propose adaptively selecting $n$ and $k$ based on the local uncertainty or entropy of tokens, enabling more aggressive leaps in low-entropy regions and finer granularity in high-entropy ones. However, such flexibility introduces challenges, such as organizing output tokens with variable leap ranges (how to decode). We consider this an avenue for future work. L-MTP serves as a sound prototype to inspire and support more dynamic leap patterns. Our formal analysis in Section 4 (Theoretical Analysis) provides principles to guide future research. We look forward to further explorations inspired by L-MTP.

W2: Regarding the effectiveness improvement of L-MTP over MTP (as shown in Table 1), the paper could provide some qualitative analysis, such as a case study, to offer more intuition behind the improvements.

Response

Thank you for your suggestion! We provide some case studies with explanations as follows. Due to the word limitation, we highlight the differences to demonstrate the effectiveness of L-MTP. L-MTP improves upon standard MTP by predicting non-consecutive, leaped tokens instead of adjacent ones, which forces the model to learn longer-range dependencies and a more structured output plan, leading to better performance. Here is a case: when solving the problem "A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?", MTP predicts the next few words sequentially, ["The", "amount", "of", "white"], making it prone to error propagation, while L-MTP could simultaneously predict key non-adjacent elements like ["total", "is", "3", "."], directly linking the input's numerical logic to the final answer structure and improving robustness and accuracy, as evidenced by its superior results on math benchmarks like GSM8K in Table 1. We will provide more cases to illustrate the effectiveness of L-MTP in our revised paper.

We authors sincerely thank you for your professional review attitude and comments. If you have other questions, we are happy to address them to polish this work.