Plato: Plan to Efficient Decode for Large Language Model Inference

We thank the reviewer for the detailed feedback and for recognizing the strengths of our work. Regarding your concerns:

• There is no actual additional waiting time overhead introduced by Plato’s design. Even in the worst case, where each node sequentially depends on the previous one, there is always a node being decoded in Plato’s pipeline. As a result, Plato’s decoding throughput in such fully sequential scenarios falls to the same level as standard autoregressive decoding, with no additional penalty.

• Thank you for bringing the concerns about our evaluation dataset to our attention. Our initial experiments intentionally followed setups aligned with previous work (such as SoT [1]), focusing on diverse question benchmarks that include nearly 300 questions across 28 types for comprehensive evaluation. To thoroughly assess Plato’s accuracy on math problems, we conducted additional experiments using the GSM-8K dataset. Specifically, we compared the accuracy performance of standard autoregressive decoding (AR), SoT, and Plato using the first 200 samples from GSM-8K, testing Phi-4 14b, Qwen2.5-32b, and Llama3.1-70b models. We employed Deepseek-V3-0324 to directly compare numeric results with ground truth answers provided by GSM-8K. Results are summarized below:

Correctness rate:

Speedup over autoregressive:

Because the steps of math problems are highly sequential, parallel opportunity is very limited, and thus the speedup of Plato over autoregressive is moderate. Still, the 1.17 × average speed‑up across the three models shows that Plato can exploit the limited parallel opportunities between steps while achieving the same level of accuracy as the autoregressive baselines, which demonstrates the effectiveness of Plato’s design. We also note that the quality of SoT is quite limited, which aligns with our evaluation in the paper.

• We appreciate your comment on the model size. Due to resource limitations, we only evaluated models up to 70B, which is a common evaluation setting within academic resources.

Thank you again for your thoughtful review and valuable suggestions. We hope that our response and revisions have adequately addressed your concerns and questions. We are more than glad to answer any further questions.

[1] Ning, Xuefei, et al. "Skeleton-of-thought: Large language models can do parallel decoding." ICLR’2024

We thank the reviewer for the positive feedback and for recognizing the novelty and significance of our work.

We appreciate your detailed summary of our contributions and your recognition of how our approach maintains semantic coherence while leveraging parallelism. Your understanding of our motivation to preserve logical and causal relationships for robust and accurate multi-node decoding aligns perfectly with our core goals.

We also appreciate your recognition of our empirical results, which highlight Plato’s ability to outperform standard autoregressive decoding and SoT in terms of both quality and efficiency.

Thank you again for your thoughtful review and for highlighting the strengths and potential impact of our work. We are open to further discussion if you have any additional questions.

We sincerely thank the reviewer for the thoughtful and detailed feedback, and we are pleased to address your concerns below.

• Regarding the naming of Plato: Thank you for highlighting your concerns about the name "Plato." The name derives partially from the phrase "Plan to Efficiently Decode," emphasizing our key contributions in task decomposition and efficiency. Additionally, it references the philosopher Plato, known for advocating structured reasoning, paralleling our approach's emphasis on logical coherence via dependency graphs. We will explicitly clarify this rationale in the final version.
• On the Design Section, Prompt Templates, and Minor Errors: We appreciate your feedback on the “Design” section and will incorporate these suggestions in the final version. Additionally, we will carefully proofread and correct the minor grammatical and literal errors identified throughout the manuscript.
• Experimental Analysis and Reproducibility: Thank you for suggesting improvements regarding reproducibility. We will add a detailed discussion section covering the specific choice of the temperature parameter used in our experiments and release the implementation code to facilitate easier reproduction. Our initial experiments intentionally followed setups aligned with previous work (such as SoT [1]), focusing on diverse question benchmarks, which include nearly 300 questions across 28 types for a comprehensive evaluation. Additionally, we have included updated GSM8K results to provide broader empirical validation on math datasets, as shown below. We compared the performance between standard autoregressive decoding, SoT, and Plato on math problems using the first 200 samples from the test set of GSM-8K. We tested Phi4-14b, Qwen2.5-32b, and Llama3.1-70b and used Deepseek-V3-0324 to check correctness. Because GSM-8K provides ground truth answers to math problems, we asked Deepseek to directly compare the numeric results and determine if the answer in the response was correct. The results are shown in the following table:

Correctness rate:

Speedup over autoregressive:

Because the steps of math problems are highly sequential, parallel opportunities are very limited, and thus the speedup of Plato over autoregressive is moderate. Still, the 1.17x average speedup across three models shows that Plato is able to exploit any possible parallel opportunities between steps while achieving the same level of accuracy comparing with autoregressive baselines, which demonstrates the effectiveness of Plato’s design. We also notice that the quality of SoT is quite limited, which aligns with our evaluation in the paper.

• Clarifying Prefill Overhead (O(N²)): We appreciate your suggestion to clarify the complexity of the prefill computation. The O(N²) complexity arises because, without our optimized global KV cache, generating the i-th node typically requires redundant prefills of previously generated nodes (1 through i-1). Thus, a naive approach incurs cumulative redundant overhead (1 + 2 + ... + N-1), leading to an O(N²) complexity. In contrast, our design avoids this by using incremental KV caching.

• Regarding nested graphs: In our framework, it is inherently impossible for an earlier node to depend on a later node and thus form a nested graph, due to the autoregressive nature of generation. During the planning phase, each node is generated in a sequential order—when an earlier node is generated, information about any subsequent (future) nodes is not yet available and therefore cannot be referenced. To further ensure system robustness, we have implemented explicit checks in our system to prevent any scenario where an earlier node depends on a later node, thus eliminating the possibility of deadlocks. Empirically, we have not observed any instances of such invalid dependencies occurring. We will clarify these points further in the revised manuscript.

• Improving Figure 3: Thank you for pointing out the clarity issues with Figure 3. We will revise this figure to improve readability, moving the legend externally and enhancing annotations, as suggested.

We thank the Reviewer for the insightful feedback. We hope that our response and revisions have adequately addressed your concerns and questions. We are more than glad to answer any further questions.

[1] Ning, Xuefei, et al. "Skeleton-of-thought: Large language models can do parallel decoding." ICLR’2024

We sincerely thank the reviewer for the thoughtful and detailed feedback, and we are pleased to address your concerns below.

• Thank you for raising the concerns regarding coding and math tasks. As discussed in the paper, Plato achieves superior performance in general tasks such as writing, common-sense reasoning, and knowledge-based question answering, which represent the main use cases for over 51% of users in the survey [1]. The relatively weaker performance of Plato on math tasks in our initial evaluation arises because the structured planning approach leads to lengthier, more detailed explanations, which do not align with the preferences of LLM-based judges. However, this does not imply that Plato generates incorrect math answers. To thoroughly assess Plato’s accuracy on math problems, we conducted additional experiments using the GSM-8K dataset. Specifically, we compared the accuracy performance of standard autoregressive decoding (AR), SoT, and Plato using the first 200 samples from GSM-8K, testing Phi-4 14b, Qwen2.5-32b, and Llama3.1-70b models. We employed Deepseek-V3-0324 to directly compare numeric results with ground truth answers provided by GSM-8K. Results are summarized below:

Correctness rate:

Speedup over autoregressive:

Because the steps of math problems are highly sequential, parallel opportunity is very limited, and thus the speedup of Plato over autoregressive is moderate. Still, the 1.17x average speedup across three models shows that Plato is able to exploit any possible parallel opportunities between steps while achieving the same level of accuracy comparing with autoregressive baselines, which demonstrates the effectiveness of Plato’s design. We also notice that the quality of SoT is quite limited, which aligns with our evaluation in the paper.

• Our initial experiments intentionally followed setups aligned with previous work (such as SoT [1]), focusing on diverse question benchmarks, which include nearly 300 questions across 28 types for a comprehensive evaluation. Additionally, as demonstrated through the GSM-8K evaluation, Plato maintains competitive accuracy even in mathematically oriented tasks. Regarding coding tasks, we acknowledge that Plato is less suitable. Plato’s decomposition strategy breaks coding tasks into smaller units interspersed with natural language explanations, making it challenging to aggregate these parts into a cohesive and functional code snippet. Autoregressive methods, which produce a single continuous coding output, remain better suited to such use cases. We will clarify this limitation explicitly in the revised manuscript.

Thank you once again for your insightful feedback. We hope that our response and revisions have adequately addressed your concerns and questions. We are more than glad to answer any further questions.

[1] Infodocket. Report: Close encounters of the AI kind: The increasingly human-like way people are engaging with language models. 2025.

[2] Ning, Xuefei, et al. "Skeleton-of-thought: Large language models can do parallel decoding." ICLR’2024