Thank you for volunteering the time and energy to review our paper – we appreciate your detailed feedback on places where our paper could be made more clear. Below are our responses to your concerns and questions.

• "it is not well substantiated to say that SAS is necessary for the capabilities required for BLiMP and GLUE. The intervention experiment in Sec 4.2, 4.3 can not support the causal relationship between them either. If there is a latent factor X that causes the better SAS, BLiMP, GLUE scores, adding the regularization term may suppress that latent factor X in addition to the SAS score. In this case, suppressing SAS also leads to worse BLiMP and GLUE scores. So that SAS is necessary for BLiMP and GLUE is not the only explanation for the observation in Sec 4.3, 4.3."
  • Thank you for highlighting these subtleties of causal reasoning. Our regularization explicitly suppresses only the maximum attention weights associated with syntactic structure – if our suppression somehow also suppresses some other latent factor X, then X is likely associated with an internal representation of syntax. Furthermore, our manipulations of SAS consistently impact the timing of both the structure onset and capabilities onset (Figure 5; Figure 18, Appendix N) – i.e., the BLiMP breakthrough always occurs shortly after the UAS spike (if the UAS spike occurs), regardless of whether we accelerate or delay the UAS spike using our SAS regularization. Taken together, our many experiments provide overwhelming evidence that SAS is causally related with the development of complex linguistic capabilities.
  • It is difficult to completely eliminate the possibility of a latent cause, but our insights about the relationship between the timing of phase transitions and the interactions between different learning strategies hold regardless of whether the underlying cause is exactly SAS or another internal structure that is strongly associated with syntax.
• "In general I think people use simplicity bias to explain some robustness issues because some spurious (unreliable/non-causal) features are simpler to learn than causal features, or say the model is doing some shortcut learning. However, it is hard to imagine that the syntactic structure is something the model shouldn’t rely on to solve any NLP problem. I think to talk about simplicity bias, the author should be more clear about the definition of “simplicity” and provide more evidence that the model’s prediction is really “biased” by that specific “simple” feature."
  • Thank you for mentioning this point – we see how it may be unclear. It is true that many studies of simplicity bias employ experiments with heuristics that are not just simplistic, but also spurious. However, generally the definitions of simplicity bias characterize only the complexity of the learner and not its use of spurious features (Dingle et al., Pérez et al., Nakkiran et al., Shah et al.). It may so happen that these simplistic heuristics are "shortcuts" and therefore non-causal features, but this is not a requirement for the definition of simplicity bias. Instead, we are merely referring to the observation that too much focus on a simplistic strategy (even an effective, non-spurious one) may distract from a more nuanced and complex learning strategy. In some cases, this more complex strategy may simply add more features on top of the simplistic strategy, rather than replacing it.
  • We have added a brief explanation at the end of Section 4.1 (in red) that also clarifies the definition of simplicity bias – thank you for helping us improve our paper. Appendix C.1 also expands upon the relevant prior work.
  • As for evidence that the model's prediction is really biased, our aim is only to show that the model's learning is biased towards learning human interpretable artifacts such as SAS, not that the predictions are biased by SAS. We show that BERT-Base naturally learns SAS both without intervention (Fig. 1) and even after limited amounts of suppression (Section 4.3.1).

Thanks for your reply.

For the first point, I totally agree your response. I would suggest you make your argument in this way in the paper. Though you claim that it's subtle, apparently the argument here is different from the main argument in the paper.

For the second point, after reading your elaboration you added at the end of Section 4.1, I think the argument is much clearer now.

Finally, I think my questions were answered. I appreciate your response.

Thank you for your careful review – we appreciate that you also found these ideas interesting and worth sharing with the community. Below are our answers to your questions:

• "I would like to know what kind of method/technique is used for encoding positional information. Is it the same as the original positional encoding in Vaswani et al. 2017 (https://arxiv.org/abs/1706.03762) or something more recent variants such as Rotary Positional Encoding (https://arxiv.org/abs/2104.09864v4)? I'm asking this because the syntactic dependency relates to positional information in the sentence. I wonder how much it affects (or does not affect) the experimental setup in this paper."
  • We use the same positional embeddings implementation as in the original BERT paper (Devlin et al., https://arxiv.org/abs/1810.04805) – i.e., fully trained absolute sinusoidal positional embeddings. It would definitely be interesting to explore how various positional embeddings affect the emergence of SAS and the subsequent capabilities onset. Although we were unable to re-run our entire suite of experiments with other positional encodings, there does exist limited evidence that language models learn internal syntax representations even without positional embeddings (such as in the ELMo model, in Hewitt and Manning) and in decoder-only models with causal attention (such as in the GPT-2 model, in Vig and Belinkov). These syntax probes are not exactly the same as the one we use, but are likely related (especially the attention-based probe in Vig and Belinkov).
• "This is a more open-ended question but I wonder whether we can observe similar breakthrough (steep drop in loss) in auto-regressive (causal / decoder-only) LMs."
  • This is an interesting question – there seems to be some empirical evidence that similar breakthroughs exist in auto-regressive LMs. For example, Fig. 2 in Kaplan et al. shows initial steep loss drops in large decoder-only Transformers, and Olsson et al. show that the steep loss drop in shallow attention-only models is associated with a break in in-context learning abilities and induction head formation.

Thank you for your feedback – we appreciate that you highlighted the novelty of the approach and the potential for stimulating more such work.

• "Do I understood correctly you equate the syntax-like attention patterns with simplicity bias, and at some point call them "simple heuristics" (section 5.1)? This is a bit confusing as in the NLP literature terminology like "simple heuristics" refers to undesirable reliance on surface lexical patterns (like bigrams), and reliance on syntax is considered the opposite of a simple heuristic. It would be good to make sure your unusual framing is not a cause of confusion to the readers."
  • Thank you for mentioning this potential point of confusion! We have added a brief clarification at the end of Section 4.1 (in red) – it is true that the definition of "simple" is relative, and one might refer to bigram patterns as simple vs. syntax as complex. For our purposes, we refer to any human interpretable artifact as simple, since it must be simple enough to be understood. In addition, syntax patterns can still be considered linguistically simple because it considers only the surface form of language, and not the semantics.
  • When we refer to "simple heuristics," we are referring only to the relative functional complexity of the strategy, as compared to the complexity of all possible functions that a neural network could encode. This excludes whether the heuristic is a desirable strategy or not. Although many studies of simplicity bias do employ experiments with heuristics that are both simplistic and spurious (and therefore undesirable), generally the definitions of simplicity bias characterize only the complexity of the learner and not its use of spurious features (Dingle et al., Pérez et al., Nakkiran et al., Shah et al.). We have also referenced other work on simplicity biases in Appendix C.1.
• "Minor doubt: since you use WSJ data for testing, why use silver Stanford parser dependencies instead of gold, converted from the manually created trees?"
  • Thank you for pointing this out – this is a minor miswording on our part that we have corrected in our paper (in red). For UAS evaluation, we indeed used the manual syntax annotations on WSJ from PTB-3, but we converted these annotations into Stanford Dependencies format using the Stanford Dependencies converter. This was done in order to align with the procedure in Clark et al. and the associated codebase.
• "Do you have any inkling of what your mystery alternative strategy may involve?"
  • We explored a couple hypotheses about the alternative strategy in Appendix M. Unfortunately the evidence did not appear strong enough to include these results in the main text, but we found limited evidence to support the hypotheses that $BERT_\text{SAS-}$ acquires unstructured n-gram statistics sooner than $BERT_\text{Base}$ does (Appendix M, Fig. 15). We also found limited evidence that while $BERT_\text{Base}$ appears to rely heavily on position information early on in training, $BERT_\text{SAS-}$ may rely more on factors other than position (such as semantic information) (Appendix M, Fig. 16).

Thank you for your review, we really appreciate that you also found the techniques innovative and the results interesting! Your feedback has also helped us clarify the writing in our paper.

• "All of the results are on a single model architecture (BERT base). On the one hand, this makes sense, since an extremely wide range of experiments are carried out. On the other hand, we don't know whether the connection between sudden drops in the loss and syntactic knowledge would apply at larger scales, with causal language modeling, etc."
  • We agree it would be valuable follow-up work to study whether these connections continue to be true with both auto-regressive LMs and larger scales. Although we were unable to re-run our entire suite of experiments using larger or more varied types of language models (due to both time and compute constraints), there does exist limited evidence that causal language models also demonstrate syntactic attention patterns (Vig and Belinkov) and that syntax learning improves with scale (Pérez-Mayos). Based on these findings, it seems reasonable to speculate that syntactic attention patterns may exhibit a similarly close relationship to the initial loss drop as in BERT-Base. However, confirming this and carefully analyzing the dynamics of this relationship is work we leave for future research.
• "There are so many experiments and interesting observations that the main paper makes very frequent reference to a plethora of appendices for more detail. This makes it a bit hard in places to figure out exactly what is being reported and what it all means. (E.g. the discussion of the Information Bottleneck was fairly hard to follow, even to someone who knows a bit about that literature.)"
  • Thank you for this feedback, we have revised the Information Bottleneck section (Section 4.1) to try to simplify and clarify its connection to the rest of the paper. It's true that there are a lot of results in this paper, and we appreciate your feedback on how to clarify it all.
• "Fig 1b: why do you think there's so much more variance in the BLiMP results than in the loss curves and UAS scores?"
  • The confidence interval we computed for the BLiMP results in Fig. 1(b) is based on variance across different BLiMP challenges (mentioned in the caption). In general, it is common for models to vary in accuracy across the different challenges (Warstadt et al.), since each tests a very specific linguistic capability. The loss curve, on the other hand, has a confidence interval computed from the variance across different examples, which we expect to be lower since the model is solving the same task for each example (masked language modeling). The UAS scores do not have a confidence interval in this particular figure (but in Fig. 3 we include confidence intervals across the three model seeds) since the implementation from Clark et al. did not admit straightforward calculations of variance. Since each example may have different syntax relations, it would be difficult to interpret the variance of UAS across different examples.
• "I'm curious about whether it matters that silver dependencies were used in regularization. Did you try any other "data-free" regularizers to see if they impact SAS similarly? E.g. since each token has one head, a regularizer that promotes sparsity of attention should implicitly promote SAS as well and vice versa."
  • We did at one point try a uniform regularizer that would uniformly regularize any attention weight corresponding to a pair of syntactically related tokens. Although this still required dependency relation labels (and is therefore not "data-free" as you asked about), it would still promote sparsity of attention for $\lambda>0$. However, we found that this uniform regularizer did not suppress SAS as effectively as our proposed regularizer. This is likely because if we try to suppress sparsity with the uniform regularizer, the model can still achieve a low loss with a mostly uniform distribution that spikes only on a single attention weight in a specific head and layer between syntactically related pairs of tokens.
• "Missing references: (i) Liu et al 2021, "Probing Across Time": https://aclanthology.org/2021.findings-emnlp.71/ . (ii) p 4: "Causal methods..." I have an idea of what works the authors have in mind, but think they should be explicitly cited here."
  • Thank you, we have added Liu et al. to our related work section in Appendix C.2. We have already cited multiple causal methods in App. C.2.1 but have also added these citations to the first paragraph of Section 4.

After reading the other reviews and the authors' response my view of the paper is largely unchanged. I am keeping my recommendation.