Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems

We thank the reviewer for the feedback. We hope our clarification below resolves your concerns.

Novelty

We agree that recursion is an established concept, and we have compared against prior work like RAO. In fact, our big sweep covers 59 other parameter sharing architectures. Nevertheless, we believe there is significant novelty in our work:

• A strict control for training compute (FLOPs): This is a critical difference from prior work, as it allows for fair comparisons. For instance, we demonstrate that the gains from the RAO strategy in MobileLLM disappear when a non-recursive baseline is trained longer to match the same compute budget.
• We believe we are the first to offer a no-regret strategy, which uses two ingredients that we propose: (1) stochastic RINS, and (2) linear adapters. This is a major contribution because with our recipe, practitioners do not lose anything by enabling RINS during training.
• To the best of our knowledge, we are the first to study the effectiveness of recursive architectures in pure vision and vision-language settings. We demonstrate significant gains in SigLIP.
• We carry out an extensive set of ablations. For example:
  • we study the impact of KV cache sharing and show that even when cache sharing is used to keep memory footprint fixed, RINS still offers performance gains (Figure 6).
  • we go beyond simple performance reporting by deriving data scaling laws for RINS, showing it improves both the scaling exponent and the irreducible asymptotic loss, proving that its benefits cannot be matched by simply training a baseline model for longer (Figure 5a).
  • We also study the optimal number of recursion rounds against model size and training compute.

In summary we do not just propose a minor variation of recursive inference. We study it rigorously on a fair compute-matched regime, resolve limitations in prior works, and compare RINS against 59 other parameter-sharing architectures.

Mechanistic Insight

Our hypothesis for why RINS works well in language was discussed at length in the paper using self-similarity (see for example Appendix D). Besides showing that it indeed works for language, we also present another evidence: vision (Appendix G). Since vision does not exhibit the same self-similar structure as language, we would not expect a language-inspired method to necessarily confer an advantage there, which we show to be the case indeed.

We do acknowledge that rigorous theoretical proofs would be helpful but it is difficult to prove results for deep neural networks. The alternative is to treat it as a scientific phenomenon: formulate hypotheses, make predictions based on our hypotheses and test those predictions experimentally, which is what we do in the paper.

Generalization

Language is a major domain and introducing methods that improve language models is an important contribution. It would be unfair to require a method inspired by the beautiful self-similar structure in language to also work for another domain that doesn’t have that structure, such as vision. As we said above, the negative results in pure image classification (Appendix G) are not a contradiction but rather supporting evidence for our hypothesis regarding self-similarity.

Besides, we demonstrate the success of RINS (beyond pure language) in multimodal systems where language is a key component. Our experiments with SigLIP-B/16, a vision-language model, show significant improvements across a wide range of tasks, including:

• +2.3% in 0-shot ImageNet accuracy (from 77.3% to 79.6%).
• Widespread gains in retrieval, cultural diversity, and multilinguality benchmarks (Tables 2, 3, 4, 5).

Deployment

This is an excellent point, and we are happy to provide clarification. We agree that latency is a limiting factor. But, memory footprint is also another major constraint (please see the discussion about why memory is a major bottleneck in Section 1 of the MobileLLM paper). For memory, a significant portion is due to the model parameters and another portion is due to the KV cache. RINS improves performance without increasing the model size. In addition, we show that it helps even when KV cache sharing is used, which keeps the memory footprint fixed.

The actual wall-clock time depends on the type of hardware. To make our results more general, we compared theoretical FLOPs in the paper. Nonetheless, we report below training-time statistics on TPUv5, which we will add. to the paper for completeness:

Note that RINS with AAB signature has a significantly better latency and memory footprint than RAO and still outperforms it significantly in terms of quality. We will add these results to the main paper.

Other Baselines

As we mention in the paper, RINS is not competing with other techniques such as CoT and repeated sampling because they can both be implemented together. In addition, CoT is specific to decoder-only architectures, while RINS can be used in encoder-only architectures, such as SigLIP.

More Inference is Better?

The claim that “more inference compute is better” so our findings aren't surprising is not accurate in our opinion. Yes, this often holds when models are trained on the same number of tokens but not necessarily when training compute FLOPs is controlled for. As we show in the paper, RAO and recursive inference in pure vision don’t help when we remove those confounding factors (even though we still do more inference compute at test time).

Partitioning the Model

As we discuss in Lines 151-154, we always partition the model depthwise into two equally sized blocks. It is a simple, straightforward rule that does not need further tuning.

Beyond 1B-parameter Models

As we argue from our scaling experiments, RINS helps models with >1B parameters if they are trained on more tokens.

But to further confirm this, we have trained a 2B parameter language model using either: no recursion (BL), RAO, and RINS with signature AAB. The baseline is pretrained on 200B tokens and models with recursion are trained on fewer tokens to match the total training FLOPs. As expected the baseline is always better than RAO while RINS outperforms the baseline; and the gap continues to increase in favor of RINS. This is in agreement with our scaling law studies. Please see the C4 validation perplexity scores below.

Unfortunately, since we cannot attach images to the rebuttal (as per the Neurips guidelines), we provide a table of results. We hope this is sufficient to demonstrate that RINS works well even with models larger than 1B parameters. We will add this to the supplementary materials of the paper.

We hope these clarifications have addressed the reviewer's concerns and clarified the novelty and significance of our findings. We will revise the paper to make our contribution clearer and take your concerns into account. We respectfully ask the reviewer to reconsider their evaluation, and we would be happy to answer any remaining questions during the discussion period.

Dear reviewer,

Thank you again for the detailed review. We kindly would like to follow up on our rebuttal. If there are any remaining questions or concerns, please let us know so we can respond to them during the discussion period. Otherwise, we would appreciate it if you could confirm if our rebuttal has satisfactorily answered your questions.

Sincerely

We thank the reviewer for the detailed feedback. We are very pleased that you have found our work insightful. Please see our response to your questions below. We would be happy to answer any further questions during the discussion period.

Theoretical Analysis

Our hypothesis for why RINS works well in language is discussed at length in the paper using self-similarity (see for example Appendix D). Besides showing that RINS indeed works for language, we also present another important evidence: vision (Appendix G). Since vision does not exhibit the same self-similar structure as language, we would not expect a language-inspired method to necessarily confer an advantage there, which we show to be the case indeed.

We do acknowledge that rigorous theoretical proofs would be helpful but it is difficult to prove results theoretically for deep neural networks. The alternative is to treat it as a scientific phenomenon: formulate hypotheses, make predictions based on our hypotheses and test those predictions experimentally, which is what we do in the paper.

Latency and Memory

The actual wall-clock time depends on the type of hardware. To make our results more general, we compared theoretical FLOPs in the main paper. Nonetheless, we report below training-time statistics on TPUv5, which we will add to the paper for completeness.

Note that RINS with A$^2$B signature has a significantly better latency and memory footprint than RAO and still outperforms it significantly in terms of quality. We will add these results to the main paper.

Model Size

As we argue from our scaling experiments, RINS helps models with >1B parameters if they are trained on more tokens.

But to further confirm this, we have trained a 2B parameter language model using either: no recursion (BL), RAO, and RINS with signature AAB. The baseline is pretrained on 200B tokens and models with recursion are trained on fewer tokens to match the total training FLOPs. As expected the baseline is always better than RAO while RINS outperforms the baseline; and the gap continues to increase in favor of RINS. This is in agreement with our scaling law studies. Please see the C4 validation perplexity scores below.

Unfortunately, since we cannot attach images to the rebuttal (as per the Neurips guidelines), we provide a table of results. We hope this is sufficient to demonstrate that RINS works well even with models larger than 1B parameters. We will add this to the supplementary materials of the paper.

Summary

We hope these clarifications have addressed the reviewer's questions and strengthened our contribution. We will revise the paper to take your questions into account. We would be happy to answer any remaining questions during the discussion period.

We thank the reviewer for the feedback. We are very pleased that you have found our work elegant and insightful. Please see our response to your questions below. We would be happy to answer any further questions during the discussion period.

Latency and Memory

The actual wall-clock time depends on the type of hardware. To make our results more general, we only compared theoretical FLOPs in the main paper. Nonetheless, we report below training-time statistics on TPUv5:

Note that RINS with A$^2$B signature has a significantly better latency and memory footprint than RAO and still outperforms it significantly in performance. We will add these results to the main paper for completeness.

Evaluation

While we do focus on perplexity (PPL) during the big sweep (where we compare 59 parameter-sharing architectures), we also evaluate RINS against popular baselines in downstream tasks. This includes:

• For language models, zero-shot evaluation on common sense reasoning tasks (Table 1).
• For SigLIP, zero-shot classification, retrieval, and cultural diversity metrics (Tables 2, 3, 5) and multilinguality (Appendix B, Table 4).
• Analysis of how RINS improves both the scaling exponent and the asymptotic performance limits (Section 5, Figure 5a).

In addition, we will also include the latency and memory footprint results above in the main paper for completeness.

Recursive Inference and Self-Similarity

We provide more details about it in Appendix D. Informally, self-similarity means that similar patterns exist at multiple scales in language (e.g. across paragraphs vs. across larger scopes). Since deeper layers encode high-level contextual information (which correspond to higher-level representations), we expect that a portion of what the early layers learn can be directly used in higher layers as well. But this is precisely what recursive inference enables.

Variance in Results

A few downstream evaluations have a large variance. In BoolQ, for example, if you compare Table 10 in MobileLLM paper, the 350M baseline gets 53.8% while the much smaller 125M baseline gets 57.5%, despite the fact that larger models tend to do better overall. This is most probably due to variance. Interestingly, recursive inference mitigates this noise as seen in Table 1 in our paper.

Summary

We hope these clarifications have addressed the reviewer's questions. We will revise the paper to make our contribution clearer and take your questions into account. If our answers are satisfactory, we respectfully ask the reviewer to update their score, and we would be happy to answer any remaining questions during the discussion period.

Thank you for the detailed response. You have addressed all my concerns. I have raised my score.

The paper would be benefit to show the real latency improvement, and more (theoretical or intuitive) insights on RINS in the revision.

We thank the reviewer for the feedback. We believe there is a misunderstanding regarding the scope and key results of our paper, and we hope our clarification below will resolve it.

The Scope of "Multimodal Systems" and the Role of Vision Experiments

We suspect that the reviewer is mixing the term “multimodal systems” with the term “all modalities”. Our claim is that RINS is effective for language and multimodal systems that process language (such as contrastive vision-language models). We never claimed that it works for all modalities.

Language is a major domain and introducing methods that improve language models is an important contribution. It would be unfair to require a method inspired by the beautiful self-similar structure in language to also work for another domain that doesn’t have that structure, such as vision.

The negative results in pure image classification (Appendix G) are not a contradiction but rather supporting evidence for our hypothesis. Since vision as a modality does not exhibit the same self-similar structure as language, we would not expect a language-inspired method to necessarily confer an advantage in pure vision.

We demonstrate the success of RINS in multimodal systems where language is a key component. Our experiments with SigLIP-B/16, a vision-language model, show significant improvements across a wide range of tasks, including:

• +2.3% in 0-shot ImageNet accuracy (from 77.3% to 79.6%).
• Widespread gains in retrieval, cultural diversity, and multilinguality benchmarks (Tables 2, 3, 4, 5).

These results strongly support our claim that RINS is a "winning path" for systems that process natural language, whether in isolation or as part of a multimodal architecture. We say it is a "winning path" because we have compared it against 59 other parameter-sharing architectures.

The Connection to "Inference Scaling"

RINS is, by definition, an inference-time scaling method. RINS scales the amount of computation at inference time by recursively applying a subset of the model's layers (e.g., using signature $A^rB$). This is achieved while keeping the model parameter count and the total training compute (FLOPs) fixed relative to baselines.

The Evaluation and PPL

Our evaluation is comprehensive and not limited to perplexity (PPL). This includes:

• For language models, zero-shot evaluation on common sense reasoning tasks (Table 1).
• For SigLIP, zero-shot classification, retrieval, multilinguality, and cultural diversity metrics.
• Analysis of how RINS improves both the scaling exponent and the asymptotic performance limits (Section 5, Figure 5a).

The Scale of Models Used in Experiments

While our initial, broad sweep of 59 different architectures was conducted on 300M and 600M parameter models for computational feasibility, we validated our key findings on larger 1B parameter models. See for example: Figure 3b, Figure 4, Figure 5b. Also, as we argue from our scaling experiments in Figure 5b, RINS helps models with >1B parameters if they are trained on more tokens.

But to verify this more, we have now trained a 2B parameter language model using either: no recursion (BL), RAO, and RINS with signature AAB. The baseline is pretrained on 200B tokens and models with recursion are trained on fewer tokens to match the total training FLOPs. As expected the baseline is always better than RAO while RINS outperforms the baseline; and the gap continues to increase in favor of RINS. This is in agreement with our scaling law studies. Please see the C4 validation perplexity scores below.

Unfortunately, since we cannot attach images to the rebuttal (as per the new Neurips guidelines), we provide a table of results. We hope this is sufficient to demonstrate that RINS works well even with models >1B parameters. We will add this to the supplementary materials of the paper.

Summary of Key Contributions

We feel it is unfortunate that most of our contributions that distinguish our work from others seem to have been overlooked in this review, including :

1. No-Regret" Strategy: We introduce stochastic RINS and show that stochastic RINS with linear adapters offer a no-regret strategy. This is a major contribution because using our recipe, one can choose not to use recursive inference at test time with no loss in performance while still having the option to use recursive inference to improve performance significantly for the same set of weights and same training FLOPs.

2. Multimodal and Vision Contexts: To our knowledge, this is the first work to systematically study the effectiveness of recursive architectures in both pure vision and vision-language settings. We demonstrate significant gains in SigLIP.

3. Memory-Efficient Inference: We study the impact of KV cache sharing and show that even when cache sharing is used to keep memory footprint fixed, RINS still offers performance gains (Figure 6).

4. Scaling Law Analysis: We go beyond simple performance reporting by deriving data scaling laws for RINS, showing it improves both the scaling exponent and the irreducible asymptotic loss, proving that its benefits cannot be matched by simply training a baseline model for longer (Figure 5a). We also study the optimal number of recursion rounds against model size and training compute.

We hope these clarifications have addressed the reviewer's concerns and clarified the novelty and significance of our findings. We will revise the paper to make our contribution clearer and take your concerns into account. We respectfully ask the reviewer to reconsider their evaluation, and we would be happy to answer any remaining questions during the discussion period.