Do Language Models Use Their Depth Efficiently?

We would like to thank the reviewer for the insightful review and for acknowledging the timeliness of our paper and the extensiveness of the experiments. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.

The layer contribution metrics used in the analysis are introduced based on intuition, but their theoretical justification is limited Can the authors provide further justification or theoretical grounding for the layer contribution metrics used in the analysis?

We are not sure what a satisfactory theoretical justification would look like in the eyes of the reviewer.

Our motivation was the following: the layers in a pre-layernorm Transformer modify the residual by adding their output to it. If this output changes, it means that the layer’s computation changed. If this happens because of skipping a previous layer, it means that the output produced by the previous layer was used as an input for this layer, because the computation relied on it.

The connection between depth utilization and task complexity is underexplored. The paper primarily evaluates tasks that could plausibly be solved by relatively shallow models. It remains unclear whether the conclusions would hold for tasks that inherently require deeper computation.

We chose the math domain because that is one of the tasks that we expect to require significant depth. Previous work, such as Sun et al. (2024): Transformer Layers as Painters, showed that GSM8K is the most sensitive to layer interventions among all tasks they tested. Could the reviewer provide a concrete task that is solvable with pretrained LLMs out of the box, and requires more depth than math? We are eager to explore such cases.

Do the findings still hold for problems that cannot be solved by shallow models and inherently require deeper computation? For example, is a 10-layer model only able to solve problems whose solutions can be represented by models with effective depth no greater than, say, 5 layers? Or would it learn to utilize depth more effectively in this case?

This is an excellent question. We suspect that if we train the models on a synthetic task requiring 10 layers without modeling the input question, the model will utilize all 10 available layers. In fact, previous work, such as Csordás et al. (2021), "The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization," has already shown this. However, in Tab 4. Appendix, the authors also showed that it is possible to learn the same task with a much shallower model. This model relies on shortcuts, yet achieves 100% IID performance, albeit with low OOD generalization performance. In toy settings, it is possible to disentangle these two cases with carefully designed synthetic data, but it is not generally possible to do so for pre-trained LLMs, which are trained on most of the Internet. This was the primary motivation of our study: can we find evidence that LLMs break down problems into subproblems in a manner consistent with what we would expect from a generalizing solution? Unfortunately, we saw no evidence for this.

Our hypothesis is that if we were to take the same task and combine it with language modeling, the model would likely use fewer layers for the task and dedicate the final layers to probability refinement. Similar effects in our preliminary study can be seen by comparing the Q and Q+A versions of the identical models in Fig 31 and Fig 32 in the Appendix: forcing the model to learn the question distribution seems to force the feature building stage to end earlier. Studying this systematically would be an interesting follow-up work.

Prior work [1] [2] suggests that CoT prompting improves reasoning by increasing the effective computational depth of LLMs. Do the conclusions of this paper contradict such findings? Or do they suggest that CoT compensates for poor depth utilization in standard forward passes?

Our view is that CoT compensates for the poor depth utilization and lack of decomposition, which are among the main reasons for their success. However, CoT has its limitations. For example, there is no guarantee that the CoT trace contains the ideal decomposition, which guarantees that the model can use a fixed depth. Also, during pretraining, there is no CoT available, forcing the model to learn fixed-depth computation for all problems it sees. This can result in shortcut behaviors that must be unlearned during the post-training phase. It is unclear if this happens successfully. We believe an ideal model should be able to perform CoT, while also performing dynamic latent computation within its layers.

Is the inefficiency in depth usage due to architectural limitations of Transformers (i.e., expressive capabilities) or to properties of the training process (e.g., optimization bias or data-induced inductive bias)?

This is an excellent question. In fact, we ask the same question while discussing the latent thinking methods in L306-308: “If the insensitivity of computation depth to input complexity is the consequence of the pretaining objective, such methods are fundamentally flawed. On the other hand, if the reason is the architecture, these approaches might provide the solution.“

We hypothesize that it is at least partially caused by the next-token prediction loss, especially on the “impredictable” parts of the data that are not the result of a systematic computation, but should produce a complicated distribution of next tokens. Even if it is caused by the training process, it may be possible to construct architectures that behave more efficiently.

We tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much.

We would like to thank the reviewer for the insightful review and for finding our discussion section inspiring. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.

… I found the argument that "the second half of the transformer models seem wasteful" a bit unsupported. …

We hypothesize that computing composite features for future predictions seems more useful than dedicating half of the layers to compute updates that are only useful for a single token. Additionally, inference typically uses nucleus sampling, discarding the tail of the probability distribution. Given that the tail is probably the most difficult to predict, it likely requires a significant amount of model capacity. It seems reasonable to assume that if this “distribution refinement” could be accomplished in fewer layers, and more layers could be devoted to computing useful features, the models could become more powerful predictors. Showing whether this computation is necessary or useful is an interesting future research direction.

I'm not fully understanding the logical relation between the claimed LLM depth inefficiency and the potential ineffectiveness of latent thinking methods. Even if non-recurrent transformers are inefficient in using deeper layers, a recurrent one pretrained from scratch may still evolve in a different way.

We agree with the reviewer that our paper does not prove that latent thinking models are ineffective. In fact, in the paper we write: “If the insensitivity of computation depth to the input complexity is the consequence of the pertaining objective, such methods are fundamentally flawed. On the other hand, if the reason is the architecture, these approaches might provide the solution.” Unfortunately, we are unaware of large-scale pretrained recurrent depth models that we can test; thus, we did what we considered to be the next best experiment: a controlled study of MoEUT vs Transformers on toy tasks.

We would also like to note that while not finding dynamic computation depth and decomposition in fixed-depth models does not invalidate recurrent thinking models, if we had found evidence that even fixed-depth non-recurrent models break down the problem dynamically in their layers, that would be a strong support for the latent reasoning models. Unfortunately, it seems that this is not the case.

We tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much.

We would like to thank the reviewer for the insightful review and for acknowledging the diversity of our experiments that point to the same conclusion and their relevance for future research. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.

For example, while Figure 8 does show a diagonal pattern, I would be hesitant to draw too strong of a conclusion from it. I find that it is not entirely clear what it means to "stretch out" the same computation over several layers.

We meant by “stretch out” that representations after the layers at the same relative position in the network map to each other the best. This means that similar computation is performed by both the deep and shallow models until the same percentile of the layers. An alternative would be to perform a chunk of computation corresponding to higher-level features that are impossible in shallow models. In such a case, these layers should exhibit high loss compared to any layer in the shallow model. We saw no evidence of such behavior.

I would just be cautious about stating that deeper models "stretch out" the same computation that shallower models perform over several layers, especially considering more formal results which provide results on the transformer depth necessary to perform certain computations (See e.g. the logarithmic dependence in Liu et al. 2023).

We agree that certain computations require deeper models. However, they can often be “solved” on a limited range of data using shortcuts that do not generalize (see e.g. Table 4 in the appendix of Csordás et al. (2021): The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalization). In LLMs, this is particularly challenging to detect due to the vast amount of training data. This was the primary motivation of our study: can we find evidence that the model adjusts its depth of computation based on problem complexity or operation ordering, or similar factors? Such a behavior would indicate decomposition and building on subresults. Unfortunately, we find no evidence for this.

The fact that MoEUT tends to make use of more depth in its computation than a non-universal transformer confirms the advantage of shared-layer models. I do not believe that this fact really confirms anything.

We will change the wording of this sentence: MoEUT tends to utilize more depth in its computation than a non-universal transformer suggests that shared-layer models might have an advantage in this regard.

The reason to believe that shared layers are better is that the knowledge can be easily transferred and reused in such models. Thus, it could be easier for the models to reuse already existing computations in the later layers, even if the gradient is dominated by the classification layer.

I believe that the text could be reformatted so that fewer references to sections and figures in the appendix are necessary.

We will try our best to improve the readability of the final version of the paper.

I believe that a better overview and explanation of the MoEUT results would have been in order.

We agree with the reviewer that more exploration of MoEUT would be helpful. However, for a fair comparison to MoEUTs, we would need a model trained on a similar scale and token budget as the LLMs we analyzed. Unfortunately, such a model doesn’t exist, and we do not have the budget to train it. Our MoEUT results aim to provide a preliminary study on a toy task, comparing it to a toy Transformer model trained on the same task.

The claimed "refinement of the current token's probability distribution" sounds like a very interesting phenomenon, and I believe that it could have been addressed in more detail.

We agree with the reviewer that the details of this process are important to understand. However, we believe this is a research project in its own right, and we have left it for future work.

As essentially every experimental analysis, this one could also benefit from a wider set of benchmarks.

We agree with the reviewer that more benchmarks are always beneficial, but we would also like to note that our paper offers a wide range of methods to test various aspects of the same phenomenon (future effect plots, integrated gradients, residual erasure). In terms of datasets, during the rebuttal period, we measured the future effects (Fig 3b.) on the Math dataset as well, and we obtained very similar results to the GSM8K in the paper. We will report this in the final version of our paper. Note that the toy datasets reported throughout various parts of the paper (arithmetic, multi-hop QA, DeepMind math) all exhibit a consistent behavior.

Additionally, I have to question its relevance, given that nowadays chain-of-thought based approaches seem to be the standard for the type of task the authors investigate.

Chain-of-thought helps with the step-by-step computation and decomposition. We think this is the main reason behind its success. However, CoT has its limitations. For example, there is no guarantee that the CoT trace contains the ideal decomposition guaranteeing that the model can use fixed depth. Also, during pretraining, there is no CoT available, forcing the model to learn fixed-depth computation for all problems it sees. This can result in shortcut behaviors that must be unlearned during the post-training phase. It is unclear if this happens successfully. We believe an ideal model should be able to perform CoT, while also performing dynamic latent computation within its layers.

… "the second half of the layers refining the current token's probability distribution" is framed somewhat negatively, could this not also be a positive property of non-universal transformers?

We hypothesize that computing composite features for future predictions seems more useful than dedicating half of the layers to compute updates that are only useful for a single token. Additionally, inference typically uses nucleus sampling, discarding the tail of the probability distribution. Given that the tail is probably the most difficult to predict, it likely requires a significant amount of model capacity. It seems reasonable to assume that if this “distribution refinement” could be accomplished in fewer layers, and more layers could be devoted to computing useful features, the models could become more powerful predictors. However, we agree with the reviewer that this is not a proven idea, and we acknowledge that the distribution refinement behavior might be beneficial in some way for the model. This is another interesting future research direction.

Do you have any formal results confirming that certain computations that you investigate are indeed impossible in shallower models?

No, we do not have such results at present. However, it seems reasonable to assume that an N-step arithmetic operation requires at least N layers to learn a faithful elementwise computation that can generalize. Note that it is always possible to learn an N-way operation that solves any operation in a few layers, but such a solution is unlikely to generalize to slightly different scenarios. Finding evidence that distinguishes between these two scenarios is one of the motivations behind our study.

Additionally, note that previous work, such as Sun et al. (2024), "Transformer Layers as Painters," demonstrated that, unlike many others, math-related tasks are sensitive to layer interventions across all datasets they tested. This is the reason why we based our investigation primarily on GSM8K.

If the reviewer is aware of a task that has been formally proven to require deep models and is zero-shot doable by LLMs, we would be excited to test it.

I must ask the authors what novel observations their analysis brings? … I am simply wondering which observation is entirely novel, given certain overlap of this work with at least that of Lad et al., 2024.

Our analysis has multiple novel factors. For example, we are not aware of any previous methods that are able to localize interactions between the layers directly, such as our interventions presented in Fig 3 and Fig 5. Also, we are not aware of any paper that studies whether the models vary the depth of their computation based on problem difficulty (Fig 7) or position in the computation graph (Fig 6, Fig 9, etc). Regarding the attention contribution, while the newest version of Lad et al, 2024 contains a more detailed analysis on attention (Fig. 6a), this was published in an update to the submission that happened after the NeurIPS deadline. Additionally, their attention analysis measures the attention value placed on the previous few tokens and does not directly assess the cross-position effect of different layers. We also show a connection between the diminishing importance of the later computation and the refinement behavior of the layers. Additionally, we are not aware of any previous work that tried to find a correspondence between layers in different depth networks.

Overall, the goal of our paper is to provide a novel perspective, showing that LLMs do not appear to perform dynamic, problem-dependent decomposition in their latent computation. We are not aware of a study focusing on this problem.

We tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much.

We would like to thank the reviewer for the insightful review and appreciating the papers insightful analysis and practical relevance. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.

Incorporating results from ... would situate the contribution more firmly within the current discourse and might even sharpen the interpretation of the phase-split finding.

We will add the discussion of these papers to the related work section and relate them to our findings. For example, Peri-LN [2] shows that their method has increased angular distance between the representations of different layers, indicating that it might use deeper layers more efficiently. However, if the reviewer is suggesting measuring the effects that such changes have on the model's behavior and our analysis, we would like to note that our study primarily focuses on pre-trained models, and we believe that a reliable signal about multi-step processing and reasoning can only be obtained from well-trained models. Thus, such a study needs to be conducted by one of the big industry labs, which can afford to systematically train a series of models with these modifications.

Architectural bias toward pre-LN models: All experiments use pre-layer-norm Transformers; it therefore remains unclear whether the conclusions extend to post-LN or other variants such as the OLMo architecture [5] or [2]. A small-scale replication on at least one non-pre-LN baseline, or a principled argument for transferability, would substantially strengthen the claim of generality.

We mainly analyze pre-LN models (x + f(LN(x))), because the majority of freely available models are pre-LN. We agree that the PostLN from the original Vaswani et al. (2017) paper, in the form of LN(x+f(x)) might have an impact on our findings and improve the feature building stages of the model. However, we are not aware of any successful LLMs using the PostLN layout. OLMo is x + LN(f(x)), which looks closer to PreLN than to PostLN; thus, the similarity in behavior is expected.

In the final version of the paper, we will add an analysis of the OLMo 2 models. We ran these experiments during the rebuttal. The differences from the current findings are minimal. OLMo 2 7B shows an interesting pattern, where a small "triangle" appears in the similarity maps, indicating that certain layers are composed with each other but collectively have no effect on the later computations. The lack of influence of the late layers remains evident in this model. OLMo2 13B is basically indistinguishable from Qwen 3 8B (Fig. 18. a). OLMo 2 32B is similar to Qwen 3 32B, with slightly better layer usage.

Unfortunately, we do not see a way to report our plots in the rebuttal without violating the rebuttal rules. If the reviewer has a suggestion, please let us know.

Interpretability of massive activations remains unexplored: Since residual-path signals can accumulate large magnitudes (as documented by Sun et al. [4]), tying the observed logit-sharpening behavior to activation growth patterns could provide a more mechanistic explanation of how and why later layers lose representational diversity.

We agree with the reviewer that one possibility for why the later layers have less effect on future computations might be that the residuals are growing. But there are alternative explanations. For example, since the gradients of the later layers might be directly dominated by the classifier layer’s gradient distributed through he residual, instead of the higher-order effects through composing layers. Exploring the detailed mechanism behind these effects is an important future direction and is probably worth its own study.

Clarity of key visualizations: Several figures—most notably the depth-score heat-map—lack fully labeled axes, color bars, or legends, forcing readers to consult the appendix for basic interpretive cues. Self-contained captions would improve accessibility.

We will improve this for the final version of the paper.

Limited evidence for alternative architectures: The MoEUT comparison relies on only one or two checkpoints. Extending the study to a broader range of shared-parameter or other architectures would give greater empirical weight to the “look beyond pure stacking” recommendation.

MoEUT served only as an outlook for possible future architectures. A more significant issue with the MoEUT experiments is that we lack the compute resources to fully train a base MoEUT model on a scale comparable to the Llama and Qwen models; thus, we cannot be certain of the reasons for the differences. We are not aware of any shared-layer pretrained model, but in the case such a model exists, we would be very interested in analyzing it.

Root cause of logit-smoothing remains open: The paper compellingly documents what happens in the back half of the network but stops short of explaining why. Some competing hypotheses are neither disentangled nor tested experimentally such as (1) the next-token objective provides diminishing gradient signal at depth, and (2) architectural bottlenecks curb the formation of new representations

Our paper aims to highlight that current LLMs often employ a fixed number of layers, regardless of problem complexity, and a significant portion of these layers fail to build reusable features. We analyze this effect in multiple ways throughout the paper (future effect plots, integrated gradients, residual erasure, depth score, etc). This analysis already barely fits in a single paper. While we agree with the reviewer that it is an important direction to determine the mechanistic reason behind the observed behaviors, we think that expecting to solve such a complex problem in a single paper is unreasonable.

We tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much.

We would like to thank you for your insightful review and appreciate that you felt that the work had thorough analysis and practical implications. Unfortunately, it is not possible to update the paper in the rebuttal period, so we are constrained to answer in text. Please see our responses below.

... evaluation relies on relatively small sample sizes and limited datasets ...

... It could strengthen the findings to include a broader range of prompts and datasets

While it is true that the main experiments are conducted on GSM8K, our other experiments involve simple arithmetics, multi-hop questions, and DeepMind Math, yielding consistent findings. We chose GSM8K based on the findings of Sun et al. (2024), "Transformer Layers as Painters," which showed that GSM8K is the most sensitive to layer interventions among all tasks they tested.

In the final version of the paper, we will include experiments in the appendix showing an equivalent of Fig 2., Fig. 3 and Fig. 4 for:

1. 50 examples: This will show that 10 examples is sufficient to see the general trend.
2. 10 examples from the MATH dataset: This will show robustness to the dataset in the math domain.

We have already conducted these future effect experiments during the rebuttal period, and we can confirm that they are consistent with the paper's findings. Unfortunately, due to the new NeurIPS rules, we don't see a way to upload these results.

... the analysis is limited to only the Llama and Qwen model families ...

We agree with the reviewer that having more results is always better. We would like to note that we already struggle with presenting all the model variants we tested in a comprehensive manner. We reported 3 sizes of both model families. Additionally, we also report results on the instruction-tuned Llama 70B (Fig. 19).

Nevertheless, in the final version of the paper, we will add an analysis of OLMo 2. The differences from the current findings are minimal. OLMo 2 7B shows an interesting pattern, where a small "triangle" appears in the similarity maps, indicating that certain layers are composed with each other but collectively have no effect on the later computations. The lack of influence of the late layers remains evident in this model. OLMo2 13B is basically indistinguishable from Qwen 3 8B (Fig. 18. a). OLMo 2 32B is similar to Qwen 3 32B, with slightly better layer usage.

The behavior of the Qwen models appears to differ somewhat from that of the Llama models (e.g., Figure 15, Figure 18e). Could the authors offer any explanations for these differences?...

We believe that the model's behavior is not only dependent on the model architecture, but also on the details of the training setup, including initialization and optimization setup. These training details are usually not public. Understanding what causes such behavior requires an extensive study, and we expect that fully understanding the effects could take multiple papers.

Out of the models we tested, Qwen 32B seems to be somewhat different from the others: instead of not using the late layers for features useful for future predictions, it doesn't use the early layers. This also shows that it is possible to have a well-performing model without most of the late layers being dominated by the loss-refining behavior. Based on Fig. 22i, we can see that the logit refinement behavior is more compressed, but consistent with the previous findings: the predictions start to improve when feature building stops (~layers >50).

Are there notable differences in layer-wise behavior across different prompts

Yes. The individual examples (prompts) exhibit different behaviors, but the general trends remain consistent. For example, Fig. 6 shows some of the variations in the processing behavior between two prompts. This is why we take the maximum over multiple different prompts. In the final version of the paper, we will include a few individual examples in the appendix as illustrations.

In the discussion in Section 5, the authors suggest that the second half of the layers might be wasteful, as these layers primarily perform distribution matching. However, could this distribution matching still be necessary, given that the prediction distribution halfway through the network remains substantially different from that of the final layer?

We agree with the reviewer that such refinement might be necessary. However, we are unsure about the number of layers necessary for this. For example, Figs. 18e and 22i,j show that Qwen 3 32B is able to perform refinement in a very few layers compared to other models (this model suffers from a different issue: not utilizing its early layers for feature building).

In general, this refinement behavior may be a consequence of the next-word prediction objective. However, we believe that architectural changes might influence the extent to which it dominates the network. It seems reasonable to assume that if this “distribution refinement” could be accomplished in fewer layers, and more layers could be devoted to computing useful features, the models could become more powerful predictors. Showing whether this computation is necessary or useful is an interesting future research direction.

We tried our best to resolve the concerns that the reviewer has raised. If the reviewer finds our response useful, please consider increasing the score. Thank you very much.