Quantifying Elicitation of Latent Capabilities in Language Models

We appreciate Reviewer jyH6’s detailed feedback and address each comment below.

1 Conceptual clarity: latent capability, elicitation, key terms

• Latent capability. We define it as a capability $f$ is latent in model $M$ if $M$ can achieve significantly above-random performance on tasks requiring f after minimal parameter updates, despite poor zero-shot performance.
• Elicitation. Given a latent capability, elicitation is any parameter‑efficient intervention that raises performance above a specified threshold without adding new capabilities.
• Pareto frontier. Set of (parameter p, accuracy a) points for which no other configuration attains ≥ a with ≤ p.
• Logistic‑in‑log‑parameter scaling. Empirical observation that accuracy vs. $\log_{10}(\text{parameter budget})$ fits a logistic curve.
• Prequential coding (MDL). Cumulative cross‑entropy on each example the first time it is seen; equivalent to online MDL codelength.

We will insert these formal definitions prior to the start of §4 in the main text in the camera‑ready.

We apologize for the confusion regarding the appendix sections. Due to NeurIPS's submission timeline, the main paper was submitted first with placeholder appendix sections (A, B, F, G), while the complete supplementary materials were uploaded during the designated supplementary material submission period one week later. All sections referenced in the paper have been fully populated in the supplementary materials, including complete experimental protocols (Appendix A), additional baselines (Appendix B), model comparison results (Appendix F), and statistical analyses (Appendix G). We understand the reviewer may have only accessed the initial submission PDF. All sections referenced in the paper have been fully populated in the supplementary materials.

2 Robustness of first‑epoch MDL approximation

To test sensitivity, we sample the top 5 performing runs for each seed and compute MDL for each of these runs. The resulting MDL frontiers differ by ≤ 2 % at all budgets. In addition, we compute Surplus Description Length (SDL) with a common threshold $\varepsilon$; SDL is far less sensitive to optimizer hyperparameters and reproduces the same logistic behavior. These results strengthen confidence that first‑epoch MDL is a reliable proxy for elicitation speed.

3 Generalizability beyond Llama models

Since submission, we have expanded our task and model coverage to include generative reasoning tasks on Qwen‑1.5 B and Qwen‑32 B. All three show sigmoid frontiers on MATH500, AIME24, and GPQA Diamond. We will include the curves in a new appendix section, Appendix K; no change to the main text claims is required.

4 Practical use cases for model safety & auditing

1. Elicitation vs. teaching. MDL/SDL provide an “information budget” view: if a fine‑tune requires > K bits to achieve the 95% frontier expected for the model size, auditors can flag that the run is likely teaching a new skill rather than eliciting an existing one.
2. Fine-tuning API budgeting. Regulators can bound the number of tunable parameters accessible to third‑party plug‑in developers; our frontier tells them how much accuracy gain that budget could plausibly unlock.
3. Practical evaluation setting. Safety auditors often fine‑tune a small head layer to probe hidden skills. Our results show that training ≤ 10 K random parameters cannot introduce a complex new skill in a 1B model but can reliably elicit existing ones (see Fig. 3). Therefore, auditors can cap the parameter budget and be confident they are measuring, not teaching.

We will add a concise discussion of these applications to §6.2 of the main text (practical and safety implications).

5 Practical discovery of latent capabilities

Fully automatic enumeration and detection of capabilities is beyond current technique, but two complementary paths exist:

1. Empirical probes and capability evaluations: use elicitation frontiers on diverse tasks (as in our study) to map where latent abilities saturate.
2. Mechanistic interpretability: neuron‑level analyses that identify features corresponding to a capability We see our framework as providing the cost axis that such feature‑discovery methods can relate to.

6 Random parameter selection vs. pruning/structured sparsity

We agree that structured selection can be more parameter‑efficient in principle. However, our goal is to measure total information required, including any search cost. Random subsets have zero selection information and therefore give a clean, reproducible upper bound on elicitation cost. §6.3 of the paper now incorporates a discussion of how future work could incorporate pruning by adding the bits needed to index the selected parameters (≈ $\log_2 \binom{P}{k}$) to the information budget.

7 Minor points

Our goal is to measure elicitation, not propose a new optimizer. Using an off‑the‑shelf PEFT method (LoRA) guarantees that the observed behavior stems from the parameter budget itself. The methodological novelty lies in (i) framing elicitation as an information‑constrained optimization problem, (ii) deriving and empirically validating a logistic frontier, and (iii) tying that frontier to MDL/SDL.

We agree localization would be valuable future work. Our objective in this paper is different: we isolate how much information is required to unlock a capability, irrespective of where that information resides. Once an information budget is established, existing localization tools (e.g., causal tracing, sparse probing) can operate within that budget. We will clarify this scope in §6.1 of the main text.

Summary

We provide clearer definitions, robustness checks for MDL, cross‑model confirmation, and concrete safety applications. We hope these additions resolve the reviewer’s concerns and appreciate the insightful suggestions.

We thank Reviewer D3HX for the detailed remarks and address each point below.

1 Are the MC results due to memorizing output format?

Evaluation invariant to formatting. For every multiple-choice dataset, we score the model by directly comparing logits of the canonical answer tokens (e.g., “A”/”B”/”C”/”D”) at the final position. We additionally ran an ablation that sums probabilities over several token variations per option (including different capitalization, synonyms, whitespace). The two procedures differ by <0.2 percentage points on average, showing that format memorization does not explain the gains.

Fine‑tuning vs. in‑context learning. With identical logit scoring, 10‑shot prompting improves the Llama 3.2 1B backbone on ARC‑Easy from 25% → 30%, whereas tuning only ~1K parameters raises accuracy to 65%. The large gap indicates that extra parameters unlock a latent skill rather than merely encoding the token template.

Generative control tasks. Post‑submission, we replicated the experiment on (i) Lichess chess-puzzles (exact optimal‑move generation) and (ii) TinyStories‑v2 (next‑token perplexity). Both tasks follow the same logistic elicitation curve, ruling out MC‑specific artifacts. For Llama 3.2 1B on Lichess chess-puzzles, 50% performance gap recovery occurs at ~100K parameters and 95% performance gap recovery occurs at ~1M parameters; TinyStories-v2 shows similar scaling but with the frontier shifted to the left at trainable parameter budgets comparable to the MC datasets, reflecting the relative simplicity of the task. Full tables will appear in the camera‑ready.

2 Relation between task difficulty and parameters required

Across all datasets, we observe a strong monotonic trend: more challenging tasks (lower zero‑shot and full‑FT performance) require a larger parameter budget to reach 95 % of the full‑fine‑tune gain. For example,

• ARC‑Challenge (Llama 3.2 1B): base ≈ 24 %, full‑FT ≈ 54 %; 95 %‑gap closeure at ~200K tunable parameters.
• ARC‑Easy (Llama 3.2 1B): base ≈ 25 %, full‑FT ≈ 76 %; same 95 %‑gap closure threshold met at ~5K parameters.

Similar patterns hold in the new generative tasks where Llama 3.2 1B needs ~100K parameters to cross 50 % performance gap recovery (PGR) on Lichess chess-puzzles. This supports the reviewer’s intuition that elicitation cost tracks intrinsic task difficulty.

3 FFN vs. Attention parameters

We have performed ablations targeting different components of the transformer block, in which we compare the performance of adapting only the self-attention layers of the transformer block (K, Q, V, O matrices), only the FFN layers (G, U, D matrices), the entire transformer block (all matrices K, Q, V, O, G, U, D), only the Q and V matrices, only K, Q, V matrices, only K, Q, V, G, U matrices, and the same set of randomly selected matrices in each transformer block layer.

We find that attention-only tuning (K, Q, V, O matrices) is consistently the most parameter‑efficient across the four multiple choice datasets. This suggests that the information needed to activate the reasoning skill is not confined to the FFN subspace. A full layer‑type analysis will be added to Appendix C.1; a deeper interpretability study is outside the scope of the present work, but we agree it would be valuable future work.

4 Broader domain coverage

Beyond the classification benchmarks in the submission, we now include:

1. Reasoning: Qwen‑1.5 B and 32 B fine‑tuned on reasoning traces and evaluated on MATH500, AIME‑24, and GPQA‑Diamond, following the procedure of Muennighoff et al. (2025).
2. Chess puzzles & story generation: see §1 above.

All new curves exhibit the same parameter/accuracy logistic, reinforcing the generality of the elicitation‑frontier concept. These results will appear in the appendix; no change to the main claims is required.

Summary

The additional analyses confirm that (i) the effect is not an artifact of response formatting, (ii) harder tasks demand more tunable parameters, and (iii) attention layers are at least as important as FFNs for elicitation. We appreciate the reviewer’s suggestions and believe the forthcoming appendix expansions will address these concerns.

We thank Reviewer fDeD for the constructive feedback.

Below we clarify (i) the choice of evaluation tasks, (ii) why we began with the multiple‑choice variant of GSM8K, and (iii) the Alpaca experiments referenced in the appendix.

1  Scope of evaluation beyond classification

Our goal is to measure how many parameters must be tuned before a latent capability already present in pre‑training emerges. We therefore began with multiple‑choice tasks, which (a) isolate the model’s internal reasoning from generation quality and (b) allow inexpensive, fine‑grained sweeps over 50–100 hyper‑parameter settings × 5 seeds for each parameter budget which are used to build each elicitation frontier.

Generative tasks added since submission (no change to conclusions; full tables will appear in the camera‑ready appendix):

• Lichess chess-puzzles: predict the optimal next chess move; accuracy measured by exact match on SAN tokens.
• TinyStories‑v2: next‑token language modelling, evaluated by test‑set perplexity.
• Qwen‑1.5 B & 32 B fine‑tuned on reasoning traces from the s1K dataset (a dataset of 1,000 problems with reasoning traces), evaluated on MATH500, AIME‑24, and GPQA‑Diamond, as in Muennighoff et al. (2025).

All three tasks exhibit the same logistic‑in‑log‑parameter scaling (tables and plots with corresponding results to be included in the camera‑ready). Example (Llama 3.2 1B on Lichess chess-puzzles):

• 50% PGR (performance gap recovery between zero-shot and full fine-tuning) at ~100K parameters (≈27% exact optimal move accuracy)
• 95% PGR at ~1M parameters (≈51% exact optimal move accuracy)
• Only 0.1% accuracy with 10-shot prompting.

TinyStories-v2, an easier task, matches the parameter counts seen for MC datasets, confirming that frontier position shifts with task difficulty and not output format.

These results strengthen the claim that the observed frontier is not an artifact of classification.

2  Why the multiple‑choice (MC) version of GSM8K?

Controlling for “teaching” vs. elicitation. The base models tested (1 B–8 B) score <5 % zero‑shot on GSM8K (CoT) and 1B models remain below 40% even with 8 in‑context examples. Tuning on the full generative GSM8K risked learning new capabilities, confounding our goal of quantifying elicitation. The binary choice format raises the zero‑shot baseline to ~50 % and avoids the possibility that models learn new mathematical reasoning skills by training directly on the CoT traces in the original GSM8K dataset; instead, the models must predict the answer without being allowed to reason beforehand. This enables clear separation between latent‑skill activation and learning.

Computational cost. Each frontier requires >2,000 fine‑tuning runs. Limiting loss/back‑prop to the answer token reduced wall‑clock time by ~40×, making the study feasible on available resources. After establishing the logistic pattern on MC tasks, we shifted additional compute to new domains (see §1 above) rather than re‑running GSM8K in generative mode.

Empirical consistency. Preliminary experiments (not in the initial submission) on GSM8K (CoT) with the 8B model show the same sigmoidal curve, but shifted right (larger parameter budget). This supports the explanation above and will be reported in the final, camera-ready version.

3  Alpaca fine‑tuning & win‑rate details

Your interpretation is correct, we are fine-tuning on Alpaca and evaluating on Alpaca-Eval. The results included in the current draft examine the win rate against the base model without finetuning, which increases with the number of parameters trained. For example, Llama 3.2 1B improves from 80% to 88% win rate as the parameter budget grows from 10³ to 10⁵ (see Appendix B Figure 5c).

Outlook

We are currently extending the study to additional model families (Qwen) and to full‑generation GSM8K (CoT). We will incorporate these results, together with complete Alpaca hyperparameter details, in the camera‑ready version while keeping the core methodology unchanged.

We hope these clarifications address your concerns and demonstrate that the reported behaviour generalises beyond classification‑only settings. We appreciate your thoughtful review and are happy to discuss further during the discussion phase.

We thank Reviewer yjAf for the thoughtful feedback and address each point in turn.

1 Typo (“of of”) and missing caption (Fig. 4)

We thank the reviewer for pointing out these edits. Both have been corrected in our working draft and will be reflected in the camera‑ready version.

2 First‑epoch MDL, memorization, and over‑fitting vs. compression

Why memorizing noise does not lower first‑epoch MDL. Prequential MDL records the loss at the moment each example is first encountered; that loss is proportional to the codelength Alice must send to Bob before either of them has updated their model on that label. Because future noise labels are independent of past noise, memorization cannot reduce subsequent surprisal, so the cumulative codelength (and thus MDL) remains high.

Relationship to over‑fitting. Over‑fitting arises in later epochs when the model re‑uses the same training examples. First‑epoch MDL is intentionally insensitive to this: it measures how quickly a fixed set of tunable parameters can be exploited to improve predictive performance, independent of eventual memorization. We therefore view first‑epoch MDL as complementary to final validation accuracy:

• Validation accuracy frontier: ultimate performance attainable with a given parameter budget.
• First‑epoch MDL frontier: facility with which the model reorients its existing representations; lower MDL ⇒ fewer bits required to communicate labels ⇒ the latent capability is easier to elicit.

Both frontiers are plotted over identical datasets and model backbones, so comparisons remain valid despite MDL’s linear growth with dataset size.

3 Cumulative MDL and SDL as a hyperparameter‑agnostic alternative

We appreciate the reviewer's suggestion to consider training dynamics beyond the first epoch. We should clarify our terminology: MDL inherently measures the codelength for transmitting the dataset labels exactly once. After the first epoch, all labels have been transmitted, so traditional MDL doesn't extend to multiple epochs.

The reviewer's insight about hyperparameter dependence is valuable and motivates our use of Surplus Description Length (SDL). While prequential MDL captures first-epoch compression, SDL extends this to measure cumulative excess loss until convergence to a specified loss threshold is reached, regardless of the number of samples required to reach this threshold. This addresses the spirit of the reviewer's suggestion by providing a hyperparameter-agnostic metric that captures the full training trajectory.

To decouple elicitation speed from learning‑rate choice, we follow Whitney et al. (2021) and compute SDL:

1. Choose a common loss threshold,  $\varepsilon$ (constant across budgets/seeds).
2. For each fine‑tuned run, accumulate the excess loss, $\text{SDL} = \sum_{t=1}^{T} \max\bigl(\ell_t - \varepsilon , 0\bigr),$ until the running validation loss first falls below $\varepsilon$.
3. Define sample complexity as the number of unique training examples seen by that point.

SDL integrates excess loss only up to the first step where validation loss falls below a common threshold  $\varepsilon$; after that point, further samples contribute zero bits. SDL is (i) independent of total dataset size when $\varepsilon$ is reached before the first epoch ends and (ii) less sensitive to learning rate, learning rate schedules, data order, or batch sizes than first-epoch MDL, providing a more hyperparameter-agnostic measure of elicitation speed. Runs that never cross $\varepsilon$ within the first pass are assigned an infinite SDL, making the frontier visually diverge at budgets where elicitation becomes impossible for the selected loss threshold $\varepsilon$. We verified that choosing $\varepsilon$ anywhere between 25 % and 75 % of the full‑fine‑tune gain leaves the logistic shape unchanged. In our experiments, SDL reproduces the same sigmoidal trend observed in the first‑epoch MDL, with the SDL frontier decreasing as the tunable‑parameter budget increases. Full SDL curves and sample‑complexity plots will be included in the appendix.

4 Broader task coverage and additional model families

Since submission we have extended the study in two directions:

1. Generative tasks: Lichess chess-puzzles (exact optimal-move accuracy) and TinyStories‑v2 (next‑token perplexity) using the same 1 B–8 B Llama backbones.
2. Verification across model families: Qwen‑1.5 B and Qwen‑32 B fine‑tuned on chain‑of‑thought reasoning traces generated from the s1K dataset (Muennighoff et al. (2025)), evaluated on MATH500, AIME‑24, and GPQA‑Diamond.

All new experiments exhibit the same logistic relationship between performance and parameter budget, reinforcing the generality of the elicitation frontier. For instance, Llama 3.2 1B on Lichess chess-puzzles achieves 50% performance gap recovery at ~100K parameters and 95% at ~1M parameters, while TinyStories-v2 shows similar scaling at lower parameter counts due to its relative simplicity. We will present these results in the camera‑ready and make raw logs available.

Outlook

We plan to release complete MDL, SDL, and accuracy frontiers for every task‑model pair in the public appendix. The code for computing both MDL and SDL variants will be open‑sourced.

These additions require no changes to the core methodology or claims of the paper.

We hope these clarifications resolve the reviewer’s concerns and appreciate the opportunity to improve the manuscript.