Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models

Thank you for this positive assessment! We will fix the noted typo.

Thank you for this positive assessment! In response, we will be clearer in the conclusion about the English focus, and make the figures and tables larger.

Regarding the 2000 samples: we iteratively sampled until we had 2000 but did not keep track of the overall sample required to get to that level. As a ballpark, we would estimate that roughly 50% of the initial entries were “good” and so we filtered ~4000 (very rough estimate) down to 2000.The left-out entries were not all false positives, but also included entries which are non-alternating.

Thank you!

Thank you for the clarification about the 2000 samples.

I stand by my initial review and think this is a good paper.

Thank you for engaging so deeply with our paper! We are glad you found our use of the controlled rearing paradigm to be compelling, and are grateful for your detailed suggestions–to which we respond below:

1. On experiment 2’s unnaturalness: We also wondered if the results could emerge from the fact that the corpora are ordered in a way that fundamentally departs from the general structure of the target language. This is exactly why we ran our perplexity vs. effects analysis, shown in Fig 4. If the overall un-naturalness is all that explained our results, then we’d have expected to see a strong correlation between perplexity and length effects. However, in Fig 4 we see that there are cases where perplexity was low but there was no length effect (swapped-datives) and also cases where perplexity was relatively higher (random first) but the length effects were non-trivially greater than those with lower perplexity (long-first). Therefore, insofar as perplexity captures general unnaturalness in the corpus, we don’t really see a strong effect. That said, we like your idea of focusing on only re-orderable constituents—something that would be of value in dependency manipulation studies more broadly. We will explore running such a study for the Camera Ready.

2. On engaging with broader debates about memorization and generalization: We agree it would be useful for the camera ready to strengthen the broader implications of this work. If you have a particular suggestion for “influential voices arguing that LLMs' abilities come in large part from copying / direct evidence”, please do let us know. Re the line of work by Carlini et al. on memorization: we do not necessarily dispute the claim that LMs can memorize verbatim. Our claim is instead that while they do memorize, there are interesting finer-grained generalizations that they can show, something that can emerge even without direct evidence in their training data. So while we agree that direct evidence is critical to LMs, it is not the full story. We agree that we can bring out these positions more and will do so in the Camera Ready.

3. On the bigger picture and relation to human processing and learning: we see controlled rearing as a powerful and intuitive method that allows us to understand fine-grained generalization behavior in statistical learners like LMs. We are, in general, optimistic about the portability of these kinds of results to humans. But we think future work will require explicitly merging these results with psycholinguistics. In particular, there have been cases where methods similar to controlled rearing have been applied to generate hypotheses about human learners by using more stringent controls, though this field is currently in its infancy. We would be happy to make this broader level connection in the future version of this paper, and we thank you for pushing us in this direction.

Thanks for this review. We are glad you found the methods interesting and the experiments careful and controlled, even if finding the paper “rather thought-un-provoking”. We agree there is more to do here around the verb-dependent biases and think it’s well-spotted that this is fertile ground. In fact, we have an interesting series of mixed effect analyses of the verb biases, which we touch on in the paper but hope to expand on in the Camera Ready.

While we agree such analyses will be interesting, we would also push back against the idea that they should be the core contribution of this paper since the controlled rearing paradigm is new enough that there is much more work to be done at this higher level of granularity, to evaluate the paradigm and understand its strengths and limitations. We see this work as setting the stage for future linguistic exploration of the kind you describe, using a well-studied linguistic phenomenon.

Re whether results could be gleaned from analyzing dependency lengths: We also agree these results could be predicted by “carefully analyzing attested dependency length effects and other work on datives”, but the core contribution is to see if we get these (predicted) results using the controlled rearing paradigm. The predictability of the results is the point, in a way. While careful analysis of dependency effects allows us to answer questions about what cues exist or what behavioral effects are present in data, the controlled rearing paradigm allows us to answer this question purely from a generalization angle. That is, given the presence of (direct/indirect) cues, is a statistical learner able to pick up on them? In that sense we think these methods are complementary.

So, while we see the proposed follow-ups as interesting, we don’t think it is a demerit of this paper that it focuses on the core phenomenon. And we would more generally advocate for not seeing it as a demerit of work, in general, if the results are “rather intuitive”. In fact, for a relatively new method in a fast-moving area, we see it as a positive!

Thank you!