The presentation of the empirical results lacks clarity and it is unclear to me what the concrete findings and especially their implications are (see questions). In particular, it is not clear to me why it is interesting to test for generalization on a task (Dyck languages) that is a subset of the training task (first-order logic) and the generality of the observed empirical phenomena across various experimental conditions has as far as I understand not been systematically investigated (see questions).

• The writing, although reasonably coherent, does not tell a clear story, and reveals the rudimentary and provisional nature of the paper's central findings. After describing the initial setup, the paper consists of a few sections of various qualitative observations, explanations and guesses about the training dynamics. Examples are numerous:
  • "we hypothesize that lower depth expressions in phase 4 and beyond exhibit higher loss because the model has fewer previous tokens to conditon on"
  • "we suspect that longer expressions my help the model narrow its distribution to the valid tokens"
  • "at phase 6, shallow expresssions still exhibit higher loss [...], which could suggest possible overfitting or memorisation for certain lengths", etc.
• All those claims:
  • are purely post-hoc, i.e. in no way pre-registered (I'm not requiring formal pre-registration here - just on the level of some a priori theory or common sense suggesting they should happen)
  • rely on authors visually inspecting noisy training curves
  • carry no attempt to further formally investigate them, suggest alternative hypotheses, repeat experiments (for different hyperparameters, optimiser settings or initialisations), form a unifying theory, or make predictions for the future
  • are at the same time vague (as to what really happened) and very specific (as to pertaining only to that specific model and training run)
• Both the form and the substance thus make it difficult to get a clear conclusion from the experiment.
• There is no attempt to formally explain or further explore the difficulty of learning different types of FOL rules (e.g. from the point of view of computational complexity)
• The central connection to grokking is dubious. The original Power et al. paper showed that training has to continue for up to 1000x times longer than until convergence, for a very small model and a very simple, purely synthetic task. Here, authors use GPT-2-small sized model, and train it on 10,000 iterations - comparing it to the implementation in the nanoGPT repository, the model is trained for 600 000 iterations by default.
• Apart from the dataset contribution, at the current shape, the paper can be taken as a case study and a point of comparison for future work.

Clarity

As noted under strengths I think the experimental design of this work is probably its largest strength. However, in some places the description of the tasks and datasets are left quite vague. For example in the FOL task, how much of the text is shown? I assume it is everything up to the final $\rightarrow$ or similar structure (so that the model has to determine the truth or appropriate conclusion given the predicates) but it is never actually said. Similarly for the Dyck datasets, how is the model actually presented with input here and what must it output?

My second critique on the clarity is the figures. Firstly the font size is quite small, but more importantly, in some cases it is very difficult to distinguish between the lines. For example, does Figure 3a) have two blue dotted lines? Considering that the blue line is one of the most important ones for the result, this lack of clarity is a hinderance. In most cases the vague captions do little to help improve the clarity of the figures. I suggest the figures be made much clearer.

Two minor points: on the readability of the examples in Table 1. I suggest spaces be added to make these examples more readable. For example with $\forall\ x\ \text{AttendParty}(x)$ (note the spacing) rather than $\forall x \text{AttendParty}(x)$. On lines 180 to 183 the statement "Furthermore, we create...test of generalization". This statement could be made clearer as it comes immediately after the Dyck grammars description (which again could use more detail on what exactly the transformer is seeing and doing). So it was a bit of a sudden change of topic.

Quality

While the experimental design is good, I have some concerns on other aspects of this work. My first concern is that the novel dataset is generated from LLMs. While I do not disagree with this on a fundamental level, little concern is given to the unreliable nature of this approach. There are many works which use generated data to help learning, but this is slightly different as in this case it is presented as ground truth and not scrutinized at all. I think this is still necessary. I do, however, feel this is mitigated by the human labelled dataset and note that in Figure 2 the performance is similar between this validation set and the generated set. The lack of consideration here still seems to be against quality to a degree.

My primary critique, however, is that I just don't see most of the results the paper claims to obtain. Firstly, it is never described how a learning transition is defined. The numbers at the top of the loss curves are not even explained in the figure captions. The biggest offender of this is phase 6: what about epoch 140 (roughly) is important on this curve? At least for the others there is something like a peak or trough nearby (even if it is debatable that this means anything like in phase 8). But phase 6 to me corresponds to no clear salient point in the loss trajectory in Figure 2. My understanding is that these phases are determined based on Figure 2, so I can ignore the lack of key points in the other figures (although this could be clearer too if I am correct), but there should be no doubt in Figure 2 what I should be seeing.

Relatedly, I am not certain the inflection point in Figure 3 corresponds to something very meaningful. This is not helped by the difficulty of separating lines but it looks like at least two trajectories are following the same path with the blue line in Figure 3a. Figure 3b then seems to remove some lines. But either way, I don't think you can conclude that the losses of shallower expression increase compared to deeper expression based off of one setting. All depths seems to follow the exact same trajectory except for m1which still seems to track very closely. Without a notion of variance for these trajectories it is tough to claim much of anything from a gap of that size which appears for only on task setting. Figure 4 has similar claims made about shallow expression exhibiting higher loss but all of these lines look overlapping to me by phase 6.

My final large concern is with the use of the Dyck grammars - again I think it is possible that the exact setup wasn't explained and I may be assuming something incorrectly - but it seems quite unreasonable to expect the model to generalize to these datasets. Evaluating FOL statements seems fundamentally different to me that evaluating the Dyck grammars. This is supported by the fact that in Figure 2 the Dyck grammars perform quite poorly. In the first paragraph on Section 4 it is then claimed that this experiment demonstrates that the model generalizes at scale. Given the relatively high loss, this seems premature.

Some smaller concerns:

• It is unclear to me how any of the results in this work relate to grokking. I'm not certain why this is brought up as none of these models grok. The connection in the "Generalization and Grokking" paragraph of the related work does not strike me as a necessary link, given the results of this work.
• Similarly the relation to double descent doesn't make sense. However this may be due to a misunderstanding. Double descent does not relate to time but rather the number of parameter in the model. So describing the phenomenon as period changes in test error, as is done on line 507, is incorrect. The x-axis for double descent is not time or epochs, it is number of model parameters.
• In Figure 7a) it is stated that copying behaviour only emerges in the first layer, but if you only consider magnitude then later layers clearly have non-negligible copying behaviour. Why are negative eigenvalues ignored here?
• For Figure 5: I cannot see how it can be claimed that the brackets are learned before any other symbol. As far as I can tell all symbols begin learning immediately and converge at similar times. The link then to phase transitions in the loss of Figure 2 seems even more dubious.
• Figure 6: this figure seems to follow completely different dynamics to the Figure 5 losses, which makes me more concerned about the connection to phases in Figure 2 - as those phases are mainly based on test data (I believe this from Line 205 - again it isn't explained fully).

I am on board with the overall aspiration for this study to bridge the gap, overcoming the trade-off that "small scale study sacrificing realism to offer mechanistic insights" and "larger scale study gaining realism but lacking mechanistic insights". However, the presented results at intermediate scale and complexity, where majority of claims are "suspicions" rather than confirmation of falsifiable hypotheses, seems to lack neither of practicality nor clear mechanistic hypothesis and testing of it. Furthermore, the writing and presentation of figures requires significant improvements to meet the standard of ICLR conference. More concretely, for example, main result figures, i.e., Figures 2-6 are all many lines of loss curves without any explanation in the caption beyond a single sentence. Also, given its scientific nature of this paper, in its paper writing, the hypotheses need to be formulated more carefully so that they're cleanly testable.