Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study

Dear Reviewer 932f,

Thank you so much for your valuable feedback!

1. Reply to Q1 in weakness.

   ---

   Our framework is grounded in long-standing perspectives from cognitive science and educational theory (e.g., [1,2,3]), which emphasize that human learning emerges through multiple complementary pathways. Drawing from these foundations, we propose a tripartite decomposition of learning ability into three cognitively motivated dimensions:

   • Learning from Instructor (LfI): (e.g., [4, 5]). Here, the key is the presence of an interactive teaching agent—the instructor—that can shape model behavior.
   • Learning from Concept (LfC): (e.g., [6]). This dimension focuses on how models use declarative conceptual knowledge to generalize across tasks.
   • Learning from Experience (LfE): (e.g., [7,8]). learning from past interactions, e.g., previous inputs, outputs, or trajectories, without explicit instruction

   Interestingly, some observed trends in our study parallel known human learning patterns—for example, the scale-emergent use of concept inputs mirrors how abstract reasoning improves with cognitive capacity [9], and the many-shot degradation resembles overload effects in human memory systems [10].

   We will clarify these connections in the revision to better highlight the cognitive grounding of our framework.

   ---

2. Reply to Q2 in weakness.

   ---

   Thank you for the insightful comment. We agree that the notion of concept in cognitive psychology is rich and multifaceted, often extending beyond explicit rules or strategies to include structured abstraction, category formation, and transfer to novel domains.

   In our work, we adopt a pragmatic and operational definition of “concept” tailored to the current capabilities of LLMs. Specifically, we focus on concise, distilled abstractions (e.g., rules, heuristics, symbolic patterns) that serve as testable approximations of conceptual input. This makes it possible to evaluate whether a model can internalize and apply high-level structure, even without explicit training.

   While our setup centers on rule-based environments, this is not meant to reduce the broader meaning of "concept." Instead, we view it as a first step toward measuring concept-driven generalization in LLMs under controlled conditions. In future work, we hope to extend this to more open-ended or human-like concept learning tasks.

   In short, although the term "concept" is shared, the operational scope differs, our use is intentionally narrow for the sake of empirical tractability, and we appreciate the reviewer’s point in encouraging a deeper connection to cognitive theory. We will clarify this distinction in the revision.

   ---

3. Reply to Q1 in Limitations.

   ---

   Thank you for the question. We use Qwen models extensively because the family includes a broad range of scales (1.5B to 72B) under a consistent architecture and training regime. This makes it uniquely well-suited for analyzing scaling effects across different learning dimensions (e.g., Fig. 2–3, 4, 5, 6).

   That said, we also include other model families, such as GPT-4o, LLaMA3, Mistral, and Phi-3—in multiple parts of the study (e.g., Sec. 4.1, Fig. 2, Sec. 6.2, Sec. 7). These comparisons show consistent trends, supporting the robustness and generality of our conclusions beyond Qwen.

   We agree that using a single fixed opponent (Qwen2.5-32B) may raise concerns about overfitting to its strategies.

   To address this, we are currently extending our evaluation by including different Player-0:

   Player0 = Phi-4

   Player1	CH	ST	SB	SM
   gpt-4o	1.00	0.90	0.65	1.00
   gpt-4o-mini	0.65	0.88	0.86	0.72
   llama-3.1-8b	0.65	0.70	0.50	0.75
   mistral-8b	0.40	0.75	0.28	0.69
   qwen2.5-7b	0.70	0.75	0.35	0.35
   qwen3-8b	0.75	0.65	0.50	0.70

   Player0 = GPT4-o-mini

   Player1	CH	ST	SB	SM
   gpt-4o	0.80	1.00	0.65	1.00
   gpt-4o-mini	0.50	0.55	0.45	0.55
   llama-3.1-8b	0.25	0.85	0.10	0.70
   mistral-8b	0.15	0.85	0.20	0.45
   qwen2.5-7b	0.45	0.70	0.25	0.35
   qwen3-8b	0.40	0.95	0.55	0.50

   Player0 = GPT4-o

   Player1	CH	ST	SB	SM
   gpt-4o	0.60	0.50	0.45	0.50
   gpt-4o-mini	0.15	0.45	0.35	0.10
   llama-3.1-8b	0.00	0.50	0.20	0.15
   mistral-8b	0.00	0.43	0.05	0.10
   qwen2.5-7b	0.25	0.53	0.05	0.05
   qwen3-8b	0.45	0.65	0.15	0.05

   We agree that relying on a single Player-0 (Qwen2.5-32B) may raise concerns about overfitting to its strategies. To address this, we conducted preliminary experiments using alternative Player-0 models: Phi-4, GPT-4o-mini, and GPT-4o. For each Player-0, we computed the average performance of each Player-1 across four environments (CH, ST, SB, SM), ranked the models accordingly, and calculated the Spearman rank correlation with the original ranking under Qwen2.5-32B.

   • Player-0 = Phi-4 → ρ = 1.000
   • Player-0 = GPT-4o-mini → ρ = 0.829
   • Player-0 = GPT-4o → ρ = 0.714

   These results show that model rankings remain largely consistent across different opponents, suggesting that our benchmark findings are not overly sensitive to the choice of Player-0. We will include this analysis in the revised Appendix.

   Thank you again for the helpful suggestion!

   ---

4. Reply to Q2 in Limitations.

   ---

   Thank you for the question. Could you kindly elaborate a bit more on this point?

   ---

Once again, we appreciate your valuable feedback and the opportunity to address your concerns.

Reference:

[1] The modularity of mind

[2] Experience as the source of learning and development

[3] Putting students on the path to learning: The case for fully guided instruction

[4] Rapid instructed task learning: A new window into the human brain's unique capacity for flexible cognitive control

[5] Rapid instruction-based task learning (RITL) in schizophrenia

[6] An Alternative to Cognitivism: Computational Phenomenology for Deep Learning

[7] Experiential learning: Experience as the source of learning and development

[8] LLM Agents Are Experiential Learners

[9] Reasoning ability is (little more than) working-memory capacity?!

[10] The magical number 4 in short-term memory: A reconsideration of mental storage capacity

Dear Reviewer 932f,

Thank you again for your valuable feedback on our submission. We hope that our responses have addressed your questions and concerns. If you have further question, we hope to take the last chance in the rest days to clarify.

Best regards!

Dear Reviewer vwyb,

Thank you so much for your valuable feedback!

1. Reply to Q1 in weakness.

   ---

   Thank you for the thoughtful question. We address both parts below:

   (1) Why we propose these three dimensionsOur framework is grounded in long-standing perspectives from cognitive science and educational theory (e.g., [1,2,3]), which emphasize that human learning emerges through multiple complementary pathways. Drawing from these foundations, we propose a tripartite decomposition of learning ability into three cognitively motivated dimensions:

   • Learning from Instructor (LfI): (e.g., [4, 5]). Here, the key is the presence of an interactive teaching agent—the instructor—that can shape model behavior.
   • Learning from Concept (LfC): (e.g., [6]). This dimension focuses on how models use declarative conceptual knowledge to generalize across tasks.
   • Learning from Experience (LfE): (e.g., [7,8]). learning from past interactions, e.g., previous inputs, outputs, or trajectories, without explicit instruction

   These categories reflect distinct types of supervision and cognitive processing. While not strictly orthogonal, they are complementary and designed to capture different sources of learnability (instructional, conceptual, experiential) that are often entangled in real-world settings.

   (2) Ensuring that improvements reflect learning, not mere input changes

   In our experimental design, we ensure that the only change across conditions is the addition of content directly related to the targeted learning ability. No other variables are introduced. The purpose is to evaluate whether models can acquire new knowledge through instruction, concept, or experience, and not due to unrelated input differences.

   Under the same setting, all models receive identical prompts except for the learning-related content, allowing us to attribute performance differences solely to the intended learning mechanism.

   We will clarify this distinction more explicitly in the revision. Thank you again for your advice!

   ---

2. Reply to Q2 in weakness.

   ---

   Thank you for the comment. We agree that explaining the underlying causes is important but also highly challenging. Our focus in this work is to identify and quantify learning behaviors across dimensions; deeper causal analysis is left as future work, and we hope our framework provides a foundation for that.

   ---

3. Reply to Q3 in weakness.

   ---

   Thank you for the valuable feedback. We agree that using a single fixed opponent (Qwen2.5-32B) may raise concerns about overfitting to its strategies.

   To address this, we are currently extending our evaluation by including different Player-0:

   Player0 = Phi-4

   Player1	CH	ST	SB	SM
   gpt-4o	1.00	0.90	0.65	1.00
   gpt-4o-mini	0.65	0.88	0.86	0.72
   llama-3.1-8b	0.65	0.70	0.50	0.75
   mistral-8b	0.40	0.75	0.28	0.69
   qwen2.5-7b	0.70	0.75	0.35	0.35
   qwen3-8b	0.75	0.65	0.50	0.70

   Player0 = GPT4-o-mini

   Player1	CH	ST	SB	SM
   gpt-4o	0.80	1.00	0.65	1.00
   gpt-4o-mini	0.50	0.55	0.45	0.55
   llama-3.1-8b	0.25	0.85	0.10	0.70
   mistral-8b	0.15	0.85	0.20	0.45
   qwen2.5-7b	0.45	0.70	0.25	0.35
   qwen3-8b	0.40	0.95	0.55	0.50

   Player0 = GPT4-o

   Player1	CH	ST	SB	SM
   gpt-4o	0.60	0.50	0.45	0.50
   gpt-4o-mini	0.15	0.45	0.35	0.10
   llama-3.1-8b	0.00	0.50	0.20	0.15
   mistral-8b	0.00	0.43	0.05	0.10
   qwen2.5-7b	0.25	0.53	0.05	0.05
   qwen3-8b	0.45	0.65	0.15	0.05

   We agree that relying on a single Player-0 (Qwen2.5-32B) may raise concerns about overfitting to its strategies. To address this, we conducted preliminary experiments using alternative Player-0 models: Phi-4, GPT-4o-mini, and GPT-4o. For each Player-0, we computed the average performance of each Player-1 across four environments (CH, ST, SB, SM), ranked the models accordingly, and calculated the Spearman rank correlation with the original ranking under Qwen2.5-32B.

   • Player-0 = Phi-4 → ρ = 1.000
   • Player-0 = GPT-4o-mini → ρ = 0.829
   • Player-0 = GPT-4o → ρ = 0.714

   These results show that model rankings remain largely consistent across different opponents, suggesting that our benchmark findings are not overly sensitive to the choice of Player-0. We will include this analysis in the revised Appendix.

   Regarding the evaluation metric: while win rate is a coarse signal, it has shown strong differentiation across models (Table 2), and aligns with our goal of evaluating whether models can learn to succeed in dynamic environments. That said, we agree that finer-grained metrics (e.g., adaptation speed, instruction adherence) could offer additional insight and are worth exploring in future versions of LearnArena.

   ---

4. Reply to Q1 in Questions.

   ---

   In our experimental design, we ensure that the only change across conditions is the addition of content directly related to the targeted learning ability. No other variables are introduced. The purpose is to evaluate whether models can acquire new knowledge through instruction, concept, or experience, and not due to unrelated input differences.

   Under the same setting, all models receive identical prompts except for the learning-related content, allowing us to attribute performance differences solely to the intended learning mechanism.

   ---

5. Reply to Q2 in Questions.

   ---

   Thank you for the question. We provide a more detailed analysis of where the breakdown occurs in Section 6.2 (Fig. 6): across multiple models, performance peaks and then drops as the number of in-context examples exceeds ~900. This trend is consistent even in larger models (e.g., Qwen2.5-14B), suggesting that current LLMs struggle to benefit from long trajectories in ICL.

   As for why this degradation occurs, we believe it is a challenging open question. Possible factors include attention bottlenecks, interference effects, or insufficient abstraction, but a full explanation remains beyond the scope of this paper and is an important direction for future work.

   We will clarify this point further in the revision.

   ---

Once again, we appreciate your valuable feedback and the opportunity to address your concerns.

Reference:

[1] The modularity of mind

[2] Experiential learning: Experience as the source of learning and development

[3] Putting students on the path to learning: The case for fully guided instruction

[4] Rapid instructed task learning: A new window into the human brain's unique capacity for flexible cognitive control

[5] Rapid instruction-based task learning (RITL) in schizophrenia

[6] An Alternative to Cognitivism: Computational Phenomenology for Deep Learning

[7] Experiential learning: Experience as the source of learning and development

[8] ExpeL: LLM Agents Are Experiential Learners

Dear Reviewer vwyb,

Thank you again for your valuable feedback on our submission. We hope that our responses have addressed your questions and concerns. If you have further question, we hope to take the last chance in the rest days to clarify.

Best regards!

Dear Reviewer vwyb,

Thank you very much for your thoughtful feedback and for raising your score. We appreciate your recognition of the clarifications and additional experiments. As suggested, we will incorporate them into the revision.

Thank you again for your constructive feedback throughout the review process.

Best regards!

Dear Reviewer pKvT, Thank you so much for your valuable feedback!

1. Reply to Q1 in Weakness:

   ---

   Thank you for the question. Our current design follows a two-stage logic: (1) isolate and validate each learning type independently using the most suitable task setup (Sections 4–6), and (2) evaluate all three jointly under a unified benchmark (Section 7, LearnArena). We intentionally avoid using a single fixed dataset (e.g., LearnArena) for all experiments, because the goal of Sections 4–6 is to independently test each learning mechanism under minimal interference from unrelated factors. Specifically:

   • In Section 4 (Learning from Instructor), we use math tasks (e.g., GSM8K) because they naturally support a tutor–learner setup. Math problems enable clear supervision (step-by-step reasoning), and allow us to utilize strong instructors (e.g., Qwen2.5-Math-72B) to deliver high-quality guided signals.
   • In Section 5 (Learning from Concept), we choose tasks where injecting structured knowledge is feasible and meaningful:
     • Competitive environments (e.g., TextArena) allow rule-based concepts to be clearly expressed.
     • LogicGrid, NaLogic focus on formal logical constraints.
     • Plan, AlfWorld, ScienceWorld, BabyAI test concept generalization in planning and agent settings, where task structure is well-defined.
   • In Section 6 (Learning from Experience), we study two types of experience:
     • On-the-fly interactions in competitive games (TextArena), where experience is generated dynamically across rounds.
     • Pre-collected trajectories as in-context examples, used to analyze passive adaptation from past instances. These choices reflect our aim to evaluate each learning type in isolation, using domains that best match the nature of the supervision (e.g., feedback, concept, or trajectory). In Section 7, we introduce LearnArena as a unified benchmark that integrates all three learning dimensions in a shared game environment. The goal here is not to isolate but to holistically evaluate general learning ability under realistic multi-agent settings. We will clarify this rationale in the revision. Thank you again for pointing this out!

   ---

2. Reply to Q2 in Weakness:

   ---

   Thank you for the thoughtful comment. We address “what is new” along three axes:

   (1) A distinct signal from existing benchmarks. Our benchmark captures learning behavior rather than static skill: system‑level Spearman correlations are low or near‑zero with GSM8K/ARC/Winogrande and only moderate with MMLU and instruction‑following sets (Fig. 10). This indicates we are measuring a capability not well covered by standard evaluations—namely, how models learn from instructor feedback, conceptual schemas, and prior experience.

   (2) A principled way to quantify learning ability. We contribute a cognitively grounded tripartite decomposition—LfI/LfC/LfE—and a relative evaluation protocol (Norm‑Acc and deltas) that isolates the effect of the learning channel from raw task skill (Secs. 4–6). We then integrate the three channels in a unified setting (Sec. 7). To our knowledge, this offers an early, reusable methodology for comparing learning ability across models, beyond absolute task scores.

   (3) Non‑obvious, learning‑specific empirical findings. Beyond “larger is better” or “interaction helps,” our analyses surface where and when learning signals help or hurt:

   • Conceptual understanding is scale-emergent: structured knowledge hurts small models but helps larger ones (Sec. 5). Few-shot effective, many-shot brittle: performance degrades when context becomes too long (~900 examples), highlighting limits in long-context learning (Fig. 6).

   (4) On model generality: why we focus on Qwen models. We use Qwen models extensively because the family includes a broad range of scales (1.5B to 72B) under a consistent architecture and training regime. This makes it uniquely well-suited for analyzing scaling effects across different learning dimensions (e.g., Fig. 2–3, 4, 5, 6). That said, we also include other model families—such as GPT-4o, LLaMA3, Mistral, and Phi-3—in multiple parts of the study (e.g., Sec. 4.1, Fig. 2, Sec. 6.2, Sec. 7). These comparisons show consistent trends, supporting the robustness and generality of our conclusions beyond Qwen. We agree that using a single fixed opponent (Qwen2.5-32B) may raise concerns about overfitting to its strategies. To address this, we are currently extending our evaluation by including different Player-0:

   Given the rebuttal word limit, we’re happy to share the detailed table during the rebuttal phase. You may also refer to "Reply to Q3 in weakness" in Reviewer vwyb’s section for details.
   

   We agree that relying on a single Player-0 (Qwen2.5-32B) may raise concerns about overfitting to its strategies. To address this, we conducted preliminary experiments using alternative Player-0 models: Phi-4, GPT-4o-mini, and GPT-4o. For each Player-0, we computed the average performance of each Player-1 across four environments (CH, ST, SB, SM), ranked the models accordingly, and calculated the Spearman rank correlation with the original ranking under Qwen2.5-32B.

   • Player-0 = Phi-4 → ρ = 1.000
   • Player-0 = GPT-4o-mini → ρ = 0.829
   • Player-0 = GPT-4o → ρ = 0.714 These results show that model rankings remain largely consistent across different opponents, suggesting that our benchmark findings are not overly sensitive to the choice of Player-0. We will include this analysis in the revised Appendix.

   Thank you again for the helpful suggestion!

   ---

3. Reply to Q3 in Weakness:

   ---

   Thank you for the helpful suggestion. We agree that a benchmark claim should come with clear dataset statistics and deeper analyses. In the paper and appendix we already include several components, and we will make them more explicit and expand them in the revision: What is already included

   • Learning dynamics & many-shot analysis: Sec. 6.2 (Fig. 6) details non-monotonic ICL behavior (initial gains, peak, then decline; sharp drop >~900 examples) and capacity effects (14B > 7B), with discussion of possible interference/attention limits.

   • Experience-based setup details: App. A.5 describes on-policy (multi-agent games) vs. off-policy (ICL) settings, sampling, repetition (20 runs/seeded), and evaluation protocol.

   • Qualitative cases: We provide step-by-step transcripts (e.g., Tak example) illustrating decision trajectories:

     Given the rebuttal word limit, we’re happy to share the detailed case study during the rebuttal phase if that would be helpful.
     

   • Benchmark stats (to be consolidated): per-environment counts and lengths, e.g.: Each environment contains 50 unique tasks, with 20 runs per task. Input lengths vary by environment, from 63.9 tokens (SpellingBee) to 839 tokens (Stratego), with the following averages: WordChains (131), SpellingBee (63.9), SpiteAndMalice (593), Tak (739), TicTacToe (166), Stratego (839), Checkers (289), and TruthAndDeception (122.9).

     We will add these to the Appendix for better clarity.

   ---

4. Reply to Q1 in Questions:

   ---

   Thank you for your question.

   (a) Long-context model evaluation:

   Yes, in our many-shot experiments (Section 6.2, Figure 6b), we evaluate Qwen2.5-7B/14B-Instruct-1M, which support up to 1M-token windows. These models are explicitly designed for long-context usage. We observe a clear non-monotonic trend: performance increases initially with more in-context examples, peaks (at 3–9 examples depending on model size), then declines. This reflects that LLMs remain few-shot learners, and many-shot generalization remains challenging even for long-context models.

   (b) Additional evaluation with GPT-4o-mini:

   Due to cost constraints (we run each experiment with 20 trials), we use GPT-4o-mini (128k context) instead of GPT-4o for additional verification. Following [1], we use 4-shot as a baseline and test up to 100 examples:

   ICL example num → performance (GPT-4o-mini):

   0: 71.2, 4: 71.4, 10: 72.8, 15: 75.2, 20: 71.2, 30: 70.8, 50: 70.0, 80: 70.6, 100: 70.6

   This result reproduces the same rise-then-fall pattern reported in [1], where performance improves up to a peak and then drops.

   (c) Consistency with [1]:

   In [1], Section 3.1, Figure 7 (left) reports Gemini 1.5 Pro on MATH-500 with ground-truth ICL examples—a setup aligned with our many-shot configuration. Performance rises as shots increase, peaks at ~125, then declines. Although [1] truncates at 500 shots, the downward trend suggests that, if extended to >1,000 shots as in our setting—performance would continue to drop, potentially falling below zero-shot and and the 4-shot baseline used in [1]. In our experiments on MagpieMath (following the setting in Section 4) , we indeed observe the same improvement → peak → decline pattern beyond 30–100 examples, with performance eventually falling below the 4-shot baseline (noting that absolute scores differ across datasets).

   Across all models—Qwen2.5-7B → Qwen2.5-14B → GPT-4o-mini → Gemini 1.5 Pro—we observe a consistent pattern:

   • Performance peaks then degrades in many-shot settings.
   • As model capability increases, the peak shifts later, and degradation happens more slowly.

   This suggests that more capable models handle longer trajectories better but are still not immune to degradation.

   We will include the GPT-4o-mini results and a direct comparison to [1] in the revised appendix. Thank you again for the thoughtful question—it helped us strengthen the analysis and clarify important empirical trends!

   ---

Once again, we appreciate your valuable feedback and the opportunity to address your concerns!

Reference:

[1] Many-Shot In-Context Learning

Dear Reviewer pKvT,

Thank you again for your valuable feedback on our submission. We hope that our responses have addressed your questions and concerns. If you have further question, we hope to take the last chance in the rest days to clarify.

Best regards!

Dear Reviewer oteN,

Thank you so much for your valuable feedback!

1. Reply to Q1 in weakness.

   ---

   Thank you for your valuable suggestion!

   Indeed, evaluating learning ability independently of domain ability is fundamentally challenging, and we agree that task performance inevitably reflects both.

   To mitigate this, in Sections 4–6, we independently evaluate each learning type, Learning from Instructor, Learning from Concept, and Learning from Experience, under controlled conditions using relative performance metrics. Specifically:

   We report learning-induced improvements as relative gains, such as Norm Acc = training accuracy divided by original model accuracy, as shown in Figure 2 and Figure 3. Alternatively, we use delta improvements, e.g., the difference between conditions with and without concept input (see Figure 4) or score improvements after concept injection (Table 1, where darker color indicates greater gain). These analyses are designed to isolate the effect of learning, not task-specific competence.

   In Section 7, we use environments under novel rules or new task dynamics, specifically designed to minimize reliance on models’ prior task-specific capabilities. This allows us to comprehensively assess general learning ability, integrating all three aspects, learning from instructor, concept, and experience. We report absolute results here to allow direct comparison across test models, making the results more practically useful.

   We agree your valuable suggestion. In the revision, we will add a relative gain table to LearnArena, comparing performance before and after learning. [Due to the NeurIPS rebuttal word limit, we’d be happy to share the table separately during the rebuttal phase if helpful.]

   ---

2. Reply to Q2 in weakness.

   ---

   Thank you for the thoughtful question.

   While many benchmarks exist for math, coding, and reasoning, they primarily assess static performance—what a model knows—rather than how a model learns. Our work aims to fill this gap by evaluating the ability to improve through instruction, conceptual abstraction, and experience.

   Learning ability is critical for real-world applications but remains underexplored. Our benchmark provides the first systematic, cognitively grounded evaluation of this capacity.

   To validate that our benchmark reflects learning ability rather than domain knowledge, we compute Spearman correlations with nine standard benchmarks (Fig. 10 in Appendix). Results show low or near-zero correlation with GSM8K, ARC, and Winogrande, and only moderate correlation with MMLU and instruction-following datasets. This suggests that our benchmark captures distinct behavioral capabilities, particularly around adaptation and abstraction.

   In short, our work complements existing evaluations by targeting a core yet underexplored dimension of model capability: learning ability.

   ---

3. Reply to Q3 in weakness.

   ---

   Thank you for raising this important point.

   Our framework is grounded in long-standing perspectives from cognitive science and educational theory (e.g., [1,2,3]), which emphasize that human learning emerges through multiple complementary pathways. Drawing from these foundations, we propose a tripartite decomposition of learning ability into three cognitively motivated dimensions:

   • Learning from Instructor (LfI): (e.g., [4, 5]). Here, the key is the presence of an interactive teaching agent—the instructor—that can shape model behavior.
   • Learning from Concept (LfC): (e.g., [6]). This dimension focuses on how models use declarative conceptual knowledge to generalize across tasks.
   • Learning from Experience (LfE): (e.g., [7,8]). learning from past interactions, e.g., previous inputs, outputs, or trajectories, without explicit instruction

   Regarding self-critique as an alternative learning form: we agree it is a promising direction, and we view it as a special case of Learning from Instructor, where the “instructor” is the model itself.

   We appreciate the suggestion and will clarify how self-critique fits into our framework in the revision.

   ---

4. Reply to Q4 in weakness.

   ---

   Thank you for the thoughtful comment. We address “what is new” along three axes:

   (1) A distinct signal from existing benchmarks. Our benchmark captures learning behavior rather than static skill: system‑level Spearman correlations are low or near‑zero with GSM8K/ARC/Winogrande and only moderate with MMLU and instruction‑following sets (Fig. 10). This indicates we are measuring a capability not well covered by standard evaluations—namely, how models learn from instructor feedback, conceptual schemas, and prior experience.

   (2) A principled way to quantify learning ability. We contribute a cognitively grounded tripartite decomposition—LfI/LfC/LfE—and a relative evaluation protocol (Norm‑Acc and deltas) that isolates the effect of the learning channel from raw task skill (Secs. 4–6). We then integrate the three channels in a unified setting (Sec. 7). To our knowledge, this offers an early, reusable methodology for comparing learning ability across models, beyond absolute task scores.

   (3) Non‑obvious, learning‑specific empirical findings. Beyond “larger is better” or “interaction helps,” our analyses surface where and when learning signals help or hurt:

   • Conceptual understanding is scale-emergent: structured knowledge hurts small models but helps larger ones (Sec. 5).

     Few-shot effective, many-shot brittle: performance degrades when context becomes too long (~900 examples), highlighting limits in long-context learning (Fig. 6).

   ---

5. Reply to Q5 in weakness.

   ---

   Thank you for the question. Our current design follows a two-stage logic: (1) isolate and validate each learning type independently using the most suitable task setup (Sections 4–6), and (2) evaluate all three jointly under a unified benchmark (Section 7, LearnArena).

   We intentionally avoid using a single fixed dataset (e.g., LearnArena) for all experiments, because the goal of Sections 4–6 is to independently test each learning mechanism under minimal interference from unrelated factors.

   Specifically:

   • In Section 4 (Learning from Instructor), we use math tasks because they naturally support a tutor–learner setup. Math problems enable clear supervision (step-by-step reasoning), and allow us to utilize strong instructors (e.g., Qwen2.5-Math-72B) to deliver high-quality guided signals.
   • In Section 5 (Learning from Concept), we choose tasks where injecting structured knowledge is feasible and meaningful:
     • Competitive environments (e.g., TextArena) allow rule-based concepts to be clearly expressed.
     • LogicGrid, NaLogic focus on formal logical constraints.
     • Plan, AlfWorld, ScienceWorld, BabyAI test concept generalization in planning and agent settings, where task structure is well-defined.
   • In Section 6 (Learning from Experience), we study two types of experience:
     • On-the-fly interactions in competitive games (TextArena), where experience is generated dynamically across rounds.
     • Pre-collected trajectories as in-context examples (ICL), used to analyze passive adaptation from past instances.

   These choices reflect our aim to evaluate each learning type in isolation, using domains that best match the nature of the supervision (e.g., feedback, concept, or trajectory).

   In Section 7, we introduce LearnArena as a unified benchmark that integrates all three learning dimensions in a shared game environment. The goal here is not to isolate but to holistically evaluate general learning ability under realistic multi-agent settings.

   We will clarify this rationale in the revision.

   ---

6. Reply to Q1 in Questions.

   ---

   Rebuttal to R6 — On the distinction between passive consumption and experience-driven adaptation

   Thank you for the insightful question. While both LfI and LfE may resemble few-shot inputs in format, they differ fundamentally in purpose, structure, and learning assumptions:

   In passive consumption (LfI), the learner receives explicit instructional content—e.g., worked-out examples, explanations, or step-by-step reasoning—generated by a strong instructor model. The goal is didactic: to teach the model how to solve the problem, much like a teacher guiding a student through a solution. The supervision is intentional, structured, and pedagogically oriented.

   In contrast, experience-driven adaptation (LfE) provides episodic traces—e.g., raw input–output pairs from prior attempts—without explicit teaching or abstraction. The learner must infer patterns, strategies, or errors on its own, resembling unsupervised or self-supervised learning. This is cognitively more demanding, as the model must decide what (if anything) is worth learning from the examples.

   In short, LfI assumes the presence of instructional intent (even in passive form), whereas LfE assumes no such guidance—the model must learn from experience, not instruction. We will clarify this distinction in the revision. We will include a clarification of this distinction in the Appendix, and we sincerely appreciate your helpful feedback.

   ---

Once again, we appreciate your valuable feedback and the opportunity to address your concerns.

Reference:

[1] The modularity of mind

[2] Experience as the source of learning and development

[3] Putting students on the path to learning: The case for fully guided instruction

[4] Rapid instructed task learning: A new window into the human brain's unique capacity for flexible cognitive control

[5] Rapid instruction-based task learning (RITL) in schizophrenia

[6] An Alternative to Cognitivism: Computational Phenomenology for Deep Learning

[7] Experiential learning: Experience as the source of learning and development

[8] LLM Agents Are Experiential Learners

Dear Reviewer U78M,

Thank you again for your valuable feedback on our submission. We hope that our responses have addressed your questions and concerns. If you have further question, we hope to take the last chance in the rest days to clarify.

Best regards!