Auxiliary task demands mask the capabilities of smaller language models

Thank you for your thoughtful and detailed comments.

The first point concerns competence/performance distinctions. As originally conceptualized, C/P distinctions were a way to justify studying the latent construct of human linguistic representations, rather than the error-prone behaviors that people often produce. As you note, initial connectionist models were celebrated for their ability to reproduce the full distribution of observed behaviors, thus reducing the need to explain away specific human behaviors as purely performance-based.

In our work here, we apply this distinction to models instead of humans – asking whether our evaluations of their manifest behaviors fail to reveal their latent knowledge (cf. Firestone 2020; Lampinen 2023). In other words, we were trying to use the C/P distinction as a way to understand the “construct validity” (to borrow a different psychological term) of LM evaluations. Perhaps it would be less confusing just to talk about observed behavior vs. latent constructs and not C/P, given the historical context you’ve brought up.

Second, as we state in Section 3.3, we don’t draw direct analogies between model size and child development. We use size/training as proxies for how LMs might gain resources that help with overcoming task demands. This may look very different for children vs LMs: e.g., children might acquire the ability to inhibit responses, and LMs might learn how to respond to prompts. Thus, the analogy between childhood and LM size is merely conceptual, but we can still make specific, testable predictions about patterns like the demand gap. We’ll clarify this in the revision.

Third, we turn to the relationship between forced choice (FC) vs production. Conditioned on a context (e.g., analogical reasoning problem), the LM’s ability to solve the task given in the context is encoded in its distribution over subsequent tokens. This distribution is the same, whether the LM is evaluated through FC or production. The sampled response might not demonstrate the LM’s abilities because of auxiliary factors that affect decoding, whereas comparisons of answer options might reveal distinctions that are present in the LM’s distribution but would not be sampled. In this sense, production and FC can target the same cognitive construct, while imposing different auxiliary demands. We will elaborate on this as space allows.

Finally, we appreciate your suggestion about math word problems. We hope to try this at some point.

Thank you for the suggestion to add statistical testing. The key effect we would like to test for is the interaction between model size and task demands, and our prediction (visualized in Figure 1) is an interaction between these two, corresponding to the “demand gap” such that smaller models are differentially affected by task demands.

A mixed effects model (in which effects can vary across groupings – in our case, model families) is most appropriate for our data. We fit the following model in R using lme4 (the standard package for mixed effects modeling) for each of the domains with binary responses (analogical reasoning, reflective reasoning, and grammaticality judgment):

glmer(correct ~ log_n_params * eval_method + (log_n_params * eval_method | model_family), family=binomial(link="logit"))

And we fit the following model for the word prediction domain:

lmer(final_word_logprob ~ log_n_params * eval_method + (log_n_params * eval_method | model_family)

The interaction between number of parameters (log_n_params) and evaluation method (eval_method) was significant for each domain except reflective reasoning.

For the training time analyses (for which there was no model grouping factor), we fit standard GLM models for the analogical reasoning and word prediction domains, respectively:

glm(correct ~ log_training_step * eval_method, family=binomial(link="logit"))

lm(final_word_logprob ~ log_training_step * eval_method)

In both models, we found a significant interaction between the number of training steps (log_training_step) and evaluation method.

We will add these analyses and results to the revision.

Thank you for your encouraging comments and helpful suggestions.

Regarding your first point, we do acknowledge that the paper is currently a bit informal in its definition of high- and low-demand tasks. This informality follows the psychology literature, which does not provide clarity on what does and doesn’t count as a task demand. However, our exploration provides an opportunity for greater clarity, for example by defining task demands as directly related to a specific evaluation as opposed to transfer to other tasks. In fact, we view our work as providing potential value to psychologists by giving a tool for identifying task demands in future: task demands are evaluation-related demands that asymmetrically impact less capable agents (whether models or humans).

Regarding your second point, we focused on these evaluation methods because they are relevant to many, if not the vast majority of current behavioral LM evaluations. Researchers have the choice to formulate their task as a prompt or a direct measurement of string probabilities, and independently, they also have the choice to evaluate the LM in a generative (production) or discriminative (forced choice) setting. In terms of cognitive constructs, we chose word prediction and grammaticality judgment because they are perhaps the clearest cases where metalinguistic prompting differs from clearly established ways of using direct measurements, but of course there are many other domains where this contrast is relevant and can lead to different results (e.g., semantic plausibility; see Kauf et al. 2024 or Hu & Levy 2023).

For the production versus forced choice contrast, we chose the analogical and reflective reasoning domains because they have been of great recent interest in LM evaluation (per recent high-profile papers on these topics that have received a great deal of interdisciplinary attention), and because each of these datasets has a well-motivated set of answer options to be used for forced-choice evaluation. We will elaborate on all these points in the revision.

Thank you for your helpful feedback and suggestions.

We will discuss the behavior of the Pythia models in more detail in the revision. We only tested a single prompt for the metalinguistic evaluation tasks for reasons of space, and agree that the results would be stronger if multiple prompts were considered. We refer readers to Hu & Levy 2023, who tested models on 3 different metalinguistic prompts, in both English and Chinese, and found similar qualitative findings (prompting underperforms probability measurements).

Regarding your questions, we are only measuring the logits assigned to answer options conditioned on the problem/prefix; the text of the options themselves are not part of the input during forced choice eval. We also appreciate your suggestion to discuss the limitations of the evaluation methods. We will add this to the revision as space allows.

You raise an interesting point about data contamination and LAMBADA. It is possible that in this setting, the low-task demand method might actually be measuring memorization rather than the target task of next-word prediction, especially for larger models with greater memorization capacities (Carlini et al., ICLR 2023). But memorization concerns also affect the high-task demand method: if a model has actually memorized the text of LAMBADA, then it should also be able to retrieve this text under the metalinguistic prompt evaluation. One could potentially argue that the presence of the prompt would make the context less similar to what has already been seen, so the memorization effect might be weaker. This is an important empirical question for future work (we would also appreciate references if this has already been studied!).

Finally, we appreciate your suggestions about error analysis and answer ranks. We hope to explore these directions as space and time allow.