Meta-Learning for Speeding Up Large Model Inference in Decentralized Environments

Thank you for your insightful comments. We have also incorporated your suggestions into the revised manuscript and provided detailed explanations for each question below. We hope that our revisions and rebuttal could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.

Thank you for pointing out that we employed a pretrained LLM for embeddings rather than focusing on novel LLM development. Our work deliberately targets what we consider an essential aspect of the LLM ecosystem: we leverage LLM-derived semantic embeddings to enable zero-shot generalization for inference acceleration, a novel capability in decentralized AI settings. The novelty of our work is not in the creation of LLMs, but in pioneering more efficient methods for their utilization.

To clarify that this is not merely a superficial application of LLMs, we present below the empirical performance of our method compared to existing baselines:

As shown, MetaInf outperforms all heuristic and learning-based baselines in both accuracy and acceleration benefit, confirming that the embeddings generated from pretrained LLMs yield practical and measurable performance gains in method selection.

We contend that optimizing how these models are deployed and run in real-world systems, characterized by heterogeneity and resource constraints, is a significant research area in itself, crucial for translating the potential of LLMs into tangible impact.

Thank you for your insightful comments. We have also incorporated your suggestions into the revised manuscript and provided detailed explanations for each question below. We hope that our revisions and rebuttal could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.

1. Limited Real-World Application Details

Due to the limited selection of hardware available at the time of writing, we did not have the resources to obtain a diverse enough configuration pool to ensure its generalizability. To assess its robustness in a zero-shot setting, we evaluated MetaInf on a held-out test set using unseen model–GPU pairs, including both MoE and large dense models that are not present in the training data on A100 and H200 hardware. Note here A100 is included in training data while H200 is not only present in the training data, it is also not present in the embedding generation model's knowledge at the time of training, RAGs were not used to enhance model's knowledge as that introduces too much latency which is detrimental to real-time deployment. The meta-learner selects an acceleration strategy, which we then compare against baseline continuous batching inference times:

From the above experiments, we argue that the MetaInf framework, while lacking some specific optimizations to make it fully compatible with real-world deployment such as more up-to-date tiny LLM and an optimized embedding generation pipeline, it demonstrates the potential for non-trivial time savings for hosting parties.

2. Computational Overhead

Thank you pointing this out, while we did not consider temporal distribution shift in our experiments, we also mark the fact that due to the insignificant cost of retraining the meta learner with updated data compared to the resources required for hosting a LLM, a straight forward fix would be to simply perform an online update of the learner. By leveraging this in addition to continual user data ingestion, MetaInf can be well adapted to new tasks and hardware. The following table demonstrates the time savings of a inference provider and the amortized time savings will be significant. We measure the time from embedding generation to selection with a separate row, note here the embedding generation is from vanilla HuggingFace implementation of Llama3.2-1B-Instruct where embedding generation efficiency can be drastically improved upon in real deployment. As it was not the focus of the paper therefore this process was not optimized yet we still can demonstrate significant savings in inference time. We here also note that this generated embedding can be saved for training purposes and thus saving embedding generation time at training and thus allowing fast rolling updates of the learner throughout deployment.

We measured that the Llama3.1-8B-Instruct model with only continuous batching over the ShareGPT dataset with 2000 prompts and 10 max new tokens per prompt, similarly with acceleration methods chosen by our MetaInf (Chunked Prefill + Prefix Caching) and we obtain a 24% performance gain from a 0.2s inference call to determine the acceleration method to be used.

5. Limitations of Historical Data and Ensuring Robustness Without Prior Knowledge

Thank you for pointing out this problem. Historical data and online adaptation - this is a very good point, we did not consider temporal distribution shift over time, a straight forward fix would be an online adaptation fix. Because the meta learner training cost is very small and online adaptation is very practical in the real world setting. By using actual user data to continuously update the meta learner, MetaInf can be well adapted to new tasks and hardware.

6. Potential Failure of Meta-Learning Under Heavy Load or Resource Constraints

Thank you for pointing out this problem. Under constraint resources, if the training data does not contain very diverse data, it can result in failure cases. However, based on our experimental experience, meta learning methods can basically successfully predict the correct results when the data distribution is relatively comprehensive (hardware, model, inference parameters). Like most AI methods, we need a clean and generalized dataset as support. From generalizability experiments, we noticed that a drastic distribution shift can cause the model to fail to generalize, however this is an inherent problem to all AI models. Lastly, we also note that the capabilities of the predictor heavily relies on the knowledge and capabilities of the embedding generation model. If the produced embedding does not contain rich and up to date information, for example, fails to know the GPU or hallucinates on the description of the model architecture, it can lead to unexpected failures. These concerns can and should be controlled and fine-tuned to specific deployment environments.

3. Lack of discussion of latest works

Thank you for pointing out this problem. We agree that we should add more related work given this is a fast-moving domain. We plan to revise these paragraph for comparing most recent related works. We have added some latest related works discussions in inference acceleration and decentralized AI systems. The content of the Introduction and related work will be updated in future versions.

There is the revised introduction part with the latest works:

The rise of large-scale models—such as large language models (LLMs) and image generation systems—has drastically increased computational demands, prompting innovation in deployment architectures (Brown et al., 2020; Ramesh et al., 2022; Zhang et al., 2025). Traditional centralized systems, though powerful, face key limitations in scalability, data security, and cost (Li et al., 2022; Dong et al., 2025). These constraints have driven interest in decentralized architectures that distribute computation across nodes to enhance efficiency, reduce latency, and protect data privacy (Belotti et al., 2019; Zeng et al., 2024; Zhang et al., 2024).

Deploying AI in distributed settings introduces distinct challenges from centralized systems, due to heterogeneous hardware, variable network latency, and dynamic load balancing requirements (Borzunov et al., 2024; Biran & Kissos, 2025; Balseiro et al., 2025). Unlike homogeneous data centers, decentralized systems span diverse devices with varying compute, memory, and communication capabilities. This heterogeneity complicates workload distribution and calls for adaptive strategies responsive to hardware availability and runtime constraints (Shen et al., 2025).

References

• Dong, H., Jiang, J., Lu, R., Luo, J., Song, J., Li, B., Shen, Y., & Wang, Z. (2025). Beyond A Single AI Cluster: A Survey of Decentralized LLM Training. arXiv preprint arXiv:2503.11023. https://arxiv.org/abs/2503.11023
• Biran, Y., & Kissos, I. (2025). Adaptive Orchestration for Large-Scale Inference on Heterogeneous Accelerator Systems Balancing Cost, Performance, and Resilience. arXiv preprint arXiv:2503.20074.
• Balseiro, S. R., Mirrokni, V. S., & Wydrowski, B. (2025). DGD-LB: Distributed gradient descent load balancing with network latency awareness. arXiv preprint arXiv:2504.10693. https://arxiv.org/abs/2504.10693
• Zhang, ZX., Wen, YB., Lyu, HQ. et al. AI Computing Systems for Large Language Models Training. J. Comput. Sci. Technol. 40, 6–41 (2025). https://doi.org/10.1007/s11390-024-4178-1
• Shen, T., Zhu, D., Zhao, Z., Wu, C., & Wu, F. (2025). Will LLMs Scaling Hit the Wall? Breaking Barriers via Distributed Resources on Massive Edge Devices. arXiv preprint arXiv:2503.08223.
• Zeng, Z., Wang, J., Yang, J., Lu, Z., Zhuang, H., & Chen, C. (2024). PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration. arXiv preprint arXiv:2406.01394.
• Zhang, M., Shen, X., Cao, J., Cui, Z., & Jiang, S. (2024). Edgeshard: Efficient llm inference via collaborative edge computing. IEEE Internet of Things Journal.

4. How does the proposed MetaInf framework handle real-time adaptation to new tasks or hardware when no prior performance data is available?

MetaInf is a meta-learning framework designed to generalize across tasks and hardware configurations. While prior performance data can enhance prediction accuracy, the system is not strictly dependent on it. In scenarios where historical data is unavailable, MetaInf operates in a cold-start mode by leveraging metadata, such as model size, input length, and device specifications to make initial predictions. Furthermore, we can rapidly generate sufficient data to adapt to new tasks or hardware setups, which only involving a few experiments. In most practical deployment environments, this represents a transient challenge rather than a fundamental limitation.

Dear Reviewers,

Thank you again for the thoughtful reviews and the opportunity to improve our submission. We hope that our revisions and responses have addressed the key concerns raised during the review process.

As the discussion period is nearing its end, please feel free to let us know if there are any additional questions or clarifications we can provide.

If appropriate, we would also appreciate any update on the status or score of our submission. We completely understand if it is still under discussion.

Best regards, The Authors

Thank you for your insightful comments. We have also incorporated your suggestions into the revised manuscript and provided detailed explanations for each question below. We hope that our revisions and rebuttal could address your concerns and that you might consider raising the rating. Please let us know if you have any further questions or require additional results.

1. Motivation for using meta-learning method

Here we performed additional experiments on NVIDIA L4 GPUs, with continuous batching turned on by default with fixed batch size of 1024 so it's solely testing for the other two acceleration methods. We notice that the for different models, each specific acceleration method behaves differently based on the model's architecture, context length and parameter size. For example, Phi-2 performs substantially worse with prefix-caching turned on while other models like Baichuan2 performs significantly better. This demonstrates the need for a predictor that understands the different attributes and can thus pick the correct method for each model and particular hardware as we proposed.

Simple Heuristic Methods Performance

We have tested against heuristics and simpler methods and these methods perform measureably worse in terms of performance in our experiments. The following table demonstrates the time savings of inference providers and the amortized time savings will be significant.

We measured that the Llama3.1-8B-Instruct model with only continuous batching over the ShareGPT dataset with 2000 prompts and 10 max new tokens per prompt, similarly with acceleration methods chosen by our MetaInf (Chunked Prefill + Prefix Caching) and we obtain a 24% performance gain from a 0.2s inference call to determine the acceleration method to be used.

2. Revise the writing to remove the excessive use of dashes that suggests AI generated text i. AI tools were used for grammar and typo check rather than fully writing the paper. The original manuscript was fully written by us. We intend to correct that in the final version, for example, the first paragraph: "The growing demand for large-scale models such as language models and image generation systems has significantly increased computational requirements. Although centralized systems remain capable, they often face challenges related to scalability, data security, and cost. These limitations have sparked interest in decentralized architectures, which distribute computation across multiple nodes to improve efficiency, reduce latency, and enhance privacy."

3. Figure number order problem Due to the limited space, we moved some tables and figures into the appendix, and the choice of Figure 3 and Figure 5 is made out of formatting necessity at the time and with the revise of the rest of the manuscript, we do intend to make sure all tables and figures are discussed in sequence and correct any typos like 'Chucked' in the final manuscript.

4. The explanation of the term SVD For the SVD (Singular Value Decomposition), it is our oversight to not include the full explanation and background for the term. We intend to correct this with the following adjustment to the manuscript: While one-hot encodings are considered in ablation, we primarily use LLM-derived embeddings that captures rich semantic information about model configurations and task metadata. To make these embeddings more tractable for lightweight meta-learners such as MetaInf, we apply Truncated Singular Value Decomposition (SVD). By projecting high-dimensional LLM embeddings onto a lower-dimensional subspace, truncated SVD reduces computational complexity and mitigates overfitting while preserving the structure and semantic variance critical for generalization.

5. Multi-objective optimization considerations We appreciate the reviewer’s suggestion to explore multi-objective optimization, particularly in balancing latency with energy consumption. While our current work focuses solely on latency, we agree this is an important extension, especially in resource-constrained or edge settings. The current architecture is amenable to incorporating such objectives, e.g., by augmenting metadata and learning trade-off-aware selection policies. We view this as a promising direction for future work.

I want to thank the authors for providing an explanation of my concerns and appreciate their efforts! The new results resolve my concerns about motivation, I highly recommend that these results be added to the motivation of the paper, which currently is lacking. I also highly encourage the authors to address the numerous writing issues present in the current manuscript.

Based on the explanation and the new results, while appreciating the efforts put in by the authors in the rebuttal phase, I have increased my score to 7.