Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers

We thank Reviewer 9L9W for their careful reading, positive feedback on our experimental scope, and for highlighting the conceptual and practical value of Canon layers. Please see our responses below.

1. Figure crowding and visual clarity (W2)
   We acknowledge that some figures are dense and will clarify/simplify visuals in future versions.

2. Variance from random seed/data (Q2)
   BLUF: Significant variance persists even when only model initialization seeds are varied, and this is consistent with our observations in Figure 8; perplexity-based evaluation may be misleading, especially for overfitting.

   • For this rebuttal, we trained models with 3 different init seeds (fixing data seed) and found that the variance remains substantial, further confirming that performance instability is not just a function of data seeds.
   • This may not be surprising given Figure 8: there, we fixed the data seeds and still observed that minor changes in model architecture can cause large fluctuations in benchmark outcomes.
   • Regarding perplexity: Figure 8 shows large variance for LAMBADA perplexity but much less for WikiText. However, low variance on benchmarks like WikiText can reflect over-fitting and may fail to distinguish meaningful architectural differences. This supports our argument that perplexity alone may not a reliable proxy for real downstream capability.
   • Overall, we think these findings highlight the importance of our synthetic playground for controlled, interpretable architectural evaluation.

3. Canon layer kernel size (Q3)
   BLUF: Kernel size 4 was chosen purely for CUDA efficiency and reproducibility; future work will explore alternatives.

   • We adopted kernel size 4 because it is the maximum efficiently supported by the causal_conv1d CUDA implementation, enabling us to rapidly scale experiments and share reproducible results.
   • Our goal is to demonstrate effectiveness and attract collaboration from SysML experts for designing even lighter and more flexible Canon variants --- after all, even random averaging works quite well, so there ought to be better implementation choices than conv1d.

4. Related work on synthetic tasks (Q1)
   BLUF: Our synthetic tasks are deliberately chosen to probe model skills most relevant for modern LLMs, and differ in both coverage and difficulty from the formal language tasks in [1]; [2] is complementary but distinct in focus.

   • [1] (Deletang et al., "Neural Networks and the Chomsky Hierarchy")

     • Many of their tasks, such as Duplicate String and Bucket Sort, are now too easy for Transformers.
     • Their Reverse String appears challenging, but only in the binary case; with broader vocabularies (as in our DEPO/BREVO), it becomes easy for Transformers. The same holds for their Parity Check and Stack Manipulation tasks—difficulty drops with more realistic input diversity.
     • Their modular arithmetic task is similar in spirit to our MANO, but we intentionally use a much larger knowledge base (23×23 table) to better stress-test "knowledge manipulation."
     • They study multi-digit addition/multiplication, but as we explain in Figure 2 and Appendix A.1, we intentionally avoid skills like long arithmetic that are better delegated to external tools (e.g., a calculator or Python interpreter). Even Llama3-70B cannot reliably compute 452352+547647=999999, and this is not a good use of LLM's parameter capacity in practice.

   • [2] (Akyürek et al., "In-Context Language Learning: Architectures and Algorithms")

     • This work advances the understanding of in-context learning by focusing on regular languages and sequence induction. Our approach is complementary: we aim to systematically decompose LLM capabilities into atomic skills directly aligned with reasoning and knowledge challenges found in real applications, rather than primarily statistical pattern induction.

   • We appreciate the suggestion and agree that a more detailed related work discussion could be valuable in future versions.

5. Real-world relevance and generalization
   BLUF: Follow-up experiments confirm the synthetic playground's practical value at scale.

   • Since submission, we have further validated this approach: Canon layers added to 1B, 3B, and 8B Transformers pretrained on 1–2T tokens consistently yield >2% MMLU improvement, and similar gains on other lm_eval_harness tasks. By the way, our LlamaCanon-8B model outperforms the commercial Llama3-8B on such benchmarks, suggesting that this is no longer a "toy setting."
   • These results, though not included in this submission (and not cited for anonymity), are now fully open-sourced and reinforce the predictive power of our synthetic benchmarking approach for real-world advances.

Thank you again for your thorough and encouraging review!

We thank Reviewer h3Cm for their detailed review, thoughtful questions, and recognition of our experimental innovation and architectural analysis. We appreciate your suggestions for clarity and deeper comparison. Please find our responses below.

1. Central contribution and main findings
   BLUF: Our two main contributions are (1) introducing Canon layers for robust horizontal information flow, and (2) establishing a synthetic pretraining playground as a systematic, interpretable benchmark for architecture design.

   • Canon layers provide a lightweight, architecture-agnostic way to boost reasoning depth and knowledge skills across diverse models.
   • The synthetic playground enables affordable, controlled, and reproducible evaluation of core architectural skills, setting a new standard for principled LLM design and comparison.

2. Clarity and emphasis
   We acknowledge the writing could be clearer and will further highlight key messages in future versions.

3. Canon layers: Novelty vs. sliding window, kernel size, and computational overhead
   BLUF: Canon layers are distinct from sliding window attention (SWA); kernel size 4 was chosen for CUDA compatibility; computational overhead is modest yet with consistent and considerable gains.

   • Sliding-window attention (SWA) requires dense matrix multiplications (d × d), making it heavy to deploy in as many places as Canon layers. Furthermore, our experiments show SWA does not work as well unless the number of heads is very large, so it may not be worth it in practice.
   • We chose kernel size 4 purely for CUDA implementation convenience (causal_conv1d), enabling rapid experimentation and reproducibility. This allowed us to quickly test the Canon layer idea and showcase its potential, with the goal of engaging SysML experts to help develop even more lightweight and CUDA-friendly variants in future work.
   • Overhead is modest: for a 1B model with Canon-ABCD, total runtime increases by ~12% (footnote 7); for larger models (e.g., 8B), the cost drops to about 3%—yet with consistent and considerable gains in performance.

4. Synthetic task relevance and real-world validation
   BLUF: Our synthetic tasks test skills that are both challenging and predictive of large-scale real-world performance.

   • Although our synthetic tasks may appear easier than real-life tasks, they can also be more challenging in certain respects. For example, extreme 8- or 16-hop retrieval scenarios almost never occur in Common Crawl, and would likely only emerge during post-training (RL). By stress-testing model architectures on these rare but demanding skills, our synthetic playground reveals limitations and strengths that real-life pretraining may not expose until much later.
   • Since submission, we have further validated this approach: Canon layers added to 1B, 3B, and 8B Transformers pretrained on 1–2T tokens consistently yield >2% MMLU improvement, and similar gains on other lm_eval_harness tasks. Our LlamaCanon-8B model even outperforms the commercial Llama3-8B, suggesting that this is no longer a "toy setting."
   • While these results are not in the submission (and not cited for anonymity), they are fully open-sourced. We hope this further reinforces that insights from our synthetic benchmarks do transfer to real-world, large-scale settings.

Thank you again for your constructive and encouraging review!

We sincerely thank Reviewer UT7i for their strong endorsement and thoughtful feedback. We are grateful for your recognition of our experimental framework and the impact of Canon layers. Please find our responses below, along with updates from our ongoing work.

1. Figures and minor typo
   BLUF: Thank you for the suggestions; we will correct the typo and work to improve figure clarity in future versions.

2. Ablation study on Canon kernel sizes and CUDA implementation
   BLUF: Our use of kernel size 4 is due to practical CUDA support; we are seeking broader collaborations for future improvements.

   • We adopted causal_conv1d with kernel size 4 because this is what the CUDA implementation efficiently supports, allowing us to rapidly test Canon layers at scale.
   • Our goal is to first demonstrate compelling results, which we believe will attract systems and hardware experts to help us develop even more efficient and flexible (possibly CUDA-friendly) alternatives for Canon layers in future.
   • Notably, we also found that simple random averaging performs surprisingly well, suggesting there are many lightweight possibilities beyond the current implementation.

3. Multi-step reasoning in BREVO and DEPO
   BLUF: We have already explored BREVO with varying depths; Canon layers consistently increase reasoning depth.

   • As shown in Figure 14 (page 23), we varied the depth parameter in BREVO and observed that Canon layers substantially increase the model’s effective reasoning depth.
   • We chose to focus the main discussion on DEPO because it provides a more direct and interpretable probe of reasoning depth, but we agree that both perspectives are valuable.

4. Real-world validation and scale
   BLUF: Our latest large-scale results confirm that synthetic playground guidance translates to real-world gains.

   • Since submission, we have trained 1B, 3B, and 8B Transformer models on 1–2T tokens each, mirroring real-world regimes.
   • Adding Canon layers to Llama consistently increases MMLU by over 2% and gives similar improvements on other lm_eval_harness tasks. Note: our trained LlamaCanon-8B even outperforms the commercial Llama3-8B, thus this is not a toy setting.
   • While these results are not included in the submission (and note cited for anonymity), they are open-sourced and strongly support your view that our synthetic framework provides practical guidance at scale.

Thank you again for your detailed and encouraging review!

We thank Reviewer bijY for highlighting the clarity and usefulness of our work, and for appreciating our systematic approach to a difficult open problem. We appreciate your thoughtful questions about real-world relevance, task sufficiency, and evaluation, and address each below.

Overall Message:
We believe your main concerns arise from healthy skepticism about whether our synthetic tasks are sufficiently predictive and representative of real-world pretraining gains. We share this concern, and have carefully designed both our tasks and experimental phases with this in mind. Please see responses below.

1. Correlation with real-world tasks
   BLUF: Our synthetic tasks already predict some real-world improvements, especially for models like Canon+NoPE and Canon+GLA, and large-scale results (including 8B models) reinforce this correlation.

   • In the paper, we show that models improved by Canon layers on synthetic tasks—such as Canon+NoPE and Canon+GLA—become highly competitive, sometimes matching or even exceeding the performance of more complex baselines (e.g., Canon+GLA approaches Mamba2).
   • Since submission, we have validated these findings at larger scale: across 1B, 3B, and 8B-parameter Llama Transformers trained with 1–2T tokens, Llama+Canon consistently improve MMLU accuracy by 2%+ on top of Llama.
   • The combined 8B result (with Canon) even outperforms Llama3-8B on standard lm_eval_harness benchmarks, with similar gains across other tasks.
   • While these new results are not included in this submission (and cannot be cited due to anonymity), they are now fully open-sourced and visible to the community. They can't be possible without the guidance from the synthetic playground designed in this paper.

2. Why these 5 tasks? Are they “optimal”?
   BLUF: We carefully chose our tasks according to "Criteria for Task Selection" (Appendix A.1) to cover core skills relevant to real-world models, but the suite is extensible.

   • As described in Appendix A.1, we put thought into selecting tasks that reflect essential skills for modern models, while avoiding redundancy or shallowness.
   • Our approach is not to claim optimality or exhaustiveness, but rather to start with a minimal, interpretable set. If a future model’s real-world gains are not predicted by these tasks, it signals the need to augment the suite (see Appendix I, future work).
   • We focus on tasks that best isolate atomic skills most relevant for downstream use.

3. How many tasks are needed to predict model quality?
   BLUF: The current 5 tasks reliably differentiate all our studied models; expansion remains future work as needed.

   • Across both synthetic and academic-scale experiments, these tasks have distinguished all studied architectures and surfaced key design insights.
   • If future advances reveal gaps, we view this as motivation for further task development—an ongoing research agenda.

4. Are synthetic results sufficient before post-training?
   BLUF: Synthetic tasks may allow us to efficiently predict which skills architectures can acquire, before costly real-world post-training.

   • First of all, most model skills are believed to originate during pretraining, making pretraining the critical stage for understanding and improving architecture design.
   • Second, to fairly compare architectures under real-world post-training (e.g., RL), all models must be pretrained at scale using exactly the same data—a process that is very costly and prone to noise at academic scale.
   • Third, our synthetic playground is designed to efficiently predict which architectures will close skill gaps (e.g., multi-hop reasoning) that real post-training would reveal, while remaining affordable, controlled, and reproducible.

In summary, we hope the reviewer sees our synthetic benchmark as a principled, extensible “starting point” for bridging controlled architecture research and real-world model advances. We appreciate your support and thoughtful questions!

'what is the guarantee that anything good on these tasks will be good on real world tasks? What is the guarantee that they are optimal tasks?'

• I was asking for a 'guarantee'
• What if your results are a coincidence or correlation and not a general phenomenon?
• Nice to hear about Llama results, but I cannot take that into account because I cannot see them and I have no reason to blindly trust. I am not being unreasonable, I am just saying that for research, one needs to see results, otherwise this could be a case for all papers that authors say they achieved some result. Also that result was not available at time of submission.
• Maybe you can think a bit more on lines of how such a guarantee can be established, if it is even possible, and how it could look like

How do you know that all we need is to measure performance on these 5 tasks and that is all to predict goodness of pre-training.

• I am convinced with your answer point 2.
• I do not like point 3 answer. 'Across both synthetic and academic-scale experiments, these tasks have distinguished all studied architectures and surfaced key design insights.' It is not a good idea to make such general statements. Across the few tasks you studied, the tasks distinguished architectures on your criteria. LLM eval itself is such a big area that choosing a few real evals also doesn't tell everything about pre-training.

'Are synthetic results sufficient before post-training?'

• 'First of all, most model skills are believed to originate during pretraining, making pretraining the critical stage for understanding and improving architecture design.' Can you give a citation? Again, this statement should be backed with empirical evidence or citations. Pretraining is a very important stage, one of the two main stages. This was more of an open-ended question. I might not have conveyed it
• 'First of all -> Firstly.

The authors have answered some of my questions, thanks, although a few remain.