When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration

Thank you for your thoughtful review and your recognition of our work. We address your concerns below:

For example, the authors find that high-performing math models often produce dense symbolic expressions when assisting humans, which can be confusing for users. However, this can be easily addressed by modifying the prompt to encourage the LLM to focus more on helping the user rather than solving the problem independently.

This is a great point, and we fully agree that prompt design plays a critical role in the effectiveness of human-AI collaboration. However, we deliberately chose not to exhaustively optimize prompts at this stage of the investigation for multiple reasons:

First: evaluating a broad set of prompting strategies combinatorially across 8 models and multiple skill hierarchies quickly becomes infeasible. For each additional prompt variant, we would require a proportional increase in participant data to preserve statistical power and control for interaction effects. Given the resource-intensive nature of our user studies, with 118 skilled humans already involved, scaling to cover a full prompt × model × task grid would have imposed prohibitive data collection and analysis costs.

Instead, we adopt a top-down approach: we evaluate models under minimal guidance in order to measure their intrinsic capacity to communicate and convey reasoning to humans. Our goal is not to optimize each model for success via prompt engineering, but rather to assess how effectively a model, as-is, can adaptively support a user when given agency in presenting its knowledge. This reflects a more ecologically valid setting where users are not necessarily prompt engineers, and where the quality of support depends on the model’s communicative competence, not on hand-tuned scaffolding. We believe this setup also aligns more closely with the real-world deployment scenario (as mentioned in Section 1) we hope to understand.

Furthermore, evaluating alternate prompting strategies post hoc risks introducing bias and overfitting to our task suite. It would allow us to “cherry-pick” effective prompts retrospectively, creating an illusion of pedagogical alignment that might not generalize beyond our specific test cases. By contrast, our current approach generalizes and enables us to surface naturally occurring shortcomings in LLM communication, such as the overuse of dense symbolic math, based in genuine participant confusion.

Our qualitative clustering of interaction behaviors and participant feedback in Section 6 already gives us valuable insights into what kinds of LLM strategies are effective or counterproductive. These empirical observations can inform future work that systematically explores prompt design with a narrowed scope, guided by evidence rather than searching from the get go in a large solution space. Our analysis could further inform model builders aiming to improve the basic collaboration skill of LMs.

Although the authors propose an intuitive conceptual framework dividing knowledge into shared, AI-exclusive, and human-exclusive regions, they do not attempt to quantitatively analyze these segments independently. While they argue such divisions are difficult to measure, a subjective rating from participants on perceived knowledge novelty or conflict could offer insights into how often LLMs provide genuinely instructive concepts.

Absolutely. We aim to provide such a measurement in section 6, where we aggregate open-ended participant feedback to shed light on specific interaction mechanisms under the umbrella of our framework in Section 3, tagged with their frequency and correlation with success.

Thank you again for your thoughtful review. We hope this additional context resolves your lingering concerns. We look forward to continuing the conversation to improve your perception of our work and clear up any additional questions!

Thank you so much for your thoughtful review and your recognition of our work. We aim to address your core concerns below:

Evaluation is confined to coding and math, leaving knowledge-transfer effectiveness in non-STEM domains unvalidated.

We agree that this is a limitation. However, we decided to constrain our study to math and coding for 2 reasons. First, math and coding are uniquely suited to automatic verification while still having reasoning-intensive, open-ended solutions. It’s relatively much more expensive and difficult to reliably grade participants’ answers for non-STEM fields. Second, these domains match our methodology exceptionally well since “solving” a coding or math problem often requires understanding something concrete about an algorithm or proof that can be more reliably tested than domains that may have more unstructured solutions. While we agree that evaluating more diverse domains, we see our contribution as providing not just an analysis of knowledge transfer, but also a general methodology that can be applied to additional domains. We hope that this contribution can help promote the study of knowledge transfer in other domains as future work.

The study does not examine how varying prompt styles or settings influences the models’ teaching effectiveness.

This is a good point, and we fully agree that prompt design plays a critical role in the effectiveness of human-AI collaboration. However, we deliberately chose not to exhaustively optimize prompts at this stage of the investigation for multiple reasons:

First: evaluating a broad set of prompting strategies combinatorially across 8 models and multiple skill hierarchies quickly becomes infeasible. For each additional prompt variant, we would require a proportional increase in participant data to preserve statistical power and control for interaction effects. Given the resource-intensive nature of our user studies, with 118 skilled humans already involved, scaling to cover a full prompt × model × task grid would have imposed prohibitive data collection and analysis costs.

Instead, we adopt a top-down approach: we evaluate models under minimal guidance in order to measure their intrinsic capacity to communicate and convey reasoning to humans. Our goal is not to optimize each model for success via prompt engineering, but rather to assess how effectively a model, as-is, can adaptively support a user when given agency in presenting its knowledge. This reflects a more ecologically valid setting where users are not necessarily prompt engineers, and where the quality of support depends on the model’s communicative competence, not on hand-tuned scaffolding. We believe this setup also aligns more closely with the real-world deployment scenario (as mentioned in Section 1) we hope to understand.

Can you plot knowledge-transfer gain against each participant’s pre-test score (or self-reported experience) to reveal whether low- or high-expertise users benefit more from LLM collaboration?

This is a great point. We have added this in the Appendix of our paper. To summarize the findings, we find it to be roughly even across skill levels, with a slight dip towards higher skilled levels (for each model).

You report scaffolding boosts low-skill users but bores high-skill ones. Could you sketch how an adaptive strategies such as concise strategy for experts, detailed steps for novices, might reduce this gap, perhaps via a brief analysis or simulation?

This is an interesting point that would be good for future work! Because these are strategies we’ve observed post-hoc, we leave experiments on these strategies to future work. A potential method for evaluating this is to utilize the same general framework our paper provides to evaluate collaborative effectiveness and also provide models pre-hoc with users’ ELO in order to contextualize model responses. We may use models that are explicitly instructed to utilize/not to utilize scaffolding as baselines.

Hopefully this additional context resolves your concerns related to domain breadth and prompting methodology. We look forward to continuing the conversation to improve your perception of our work and clear up any additional questions!

Glad we were able to clear up your previous concerns.

Although the revised paper will include knowledge-transfer gains relative to each participant’s pre-test score, would it be possible to present this information here in a tabular or other structured format for the reviewers’ reference?

Yes, we can absolutely present this information here. Because every task was calibrated to lie one fixed Elo step above each participant’s own Elo rating, and our Elo ratings are fitted with a Bradley–Terry logistic model, we can analytically convert that fixed Elo gap into an expected unaided (“H”) success probability, 14.2 % for code and 8.1 % for math, without requiring the participant to solve additional baseline problems. Collaboration accuracy (“H + M”) therefore already measures how much additional knowledge (knowledge transfer) each model contributes. For completeness, we now present these data in tabular form in Table 2, explicitly showing the solo baseline and subtracting it from every collaborative score.

In fact, one of the paper’s key contributions is the above experimental framework for quantifying knowledge transfer. By leveraging a Bradley–Terry Elo calibration, we can predict each participant’s solo success probability for any task directly from its difficulty rating. This means we no longer need participants to tackle separate benchmark problems to establish a baseline; the model supplies that estimate “for free.” As a result, we obtain an accurate measure of human-only performance without extra trials, making the study both sample-efficient and cost-effective while still yielding a calibrated knowledge-transfer signal.

Hope this was helpful! If you have any other questions, concerns, or requests that would help us improve your understanding and perception of our work, we would be happy to provide.

Thank you for including this critical piece of information, which helps readers to understand how well the knowledge is transferred to human. Please make sure to also include the this updated table in your paper. I will raise the significance score while maintaining my current rating, with a slight inclination towards acceptance.

Thank you for your positive review of our paper! We are glad to hear you find our experimental setup “well executed” and our results interesting as well. We aim address your remaining concerns below:

Randomising (or controlling for) the response-style of LLMs may help to make some claims (such as the effect of adaptive scaffolding) stronger.

We agree, but we do see a give and take here. We believe that the response-style is a choice that the LLMs themselves should be able to make, as an LLM’s ability to select how to convey information is an important factor in its knowledge transfer abilities. Although it may make some claims stronger, controlling for the response style would remove this component, which may limit the generalizability of our study.

Could the authors show the distribution of some qualitative findings across models? For example, do models like Gemini-2.5-Pro fail in knowledge transfer due to "style issues"?

Yes: we do include a small amount of analysis on this in Section 6. We don’t dive into it in too much detail because the qualitative findings are only really able to be analyzed through participants that provided actual feedback after completing a problem: which is not all of the interactions. Therefore, such a metric would be skewed in the types of participants (via skill level) that it represents, which may paint an inaccurate picture.

Do the authors see some relationship with respect to the recent literature on the impact of LLMs on education?

Absolutely. Along with the recent results assessing the cognitive debt of LLM usage on writing tasks and learning [1] , we believe it to be incredibly important to properly isolate and evaluate the ability of humans to properly internalize LLM reasoning, as well as LLM ability to properly express reasoning in understandable terms. This knowledge transfer is core to the utility of LLMs in educative contexts: yet despite this, few works have tackled this question.

Hopefully this additional context resolves your lingering questions. Thank you again for your recognition of our work!

References: [1] Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X. H., Beresnitzky, A. V., ... & Maes, P. (2025). Your brain on chatgpt: Accumulation of cognitive debt when using an ai assistant for essay writing task. arXiv preprint arXiv:2506.08872.

Thank you for your thoughtful review! We address your concerns below:

Without a human-human control group, how can you be certain that your findings are specific to Human-AI knowledge transfer…

Below we clarify why a human-human control group, while interesting independently, is not essential for the claims made in this paper.

1. Research focus and claim scope

Our goal is to measure the relationship between LLMs’ solo capabilities and their ability to transfer / communicate their knowledge to humans as these abilities improve. We believe that understanding this relationship is primarily important because LLMs aren’t human, so as we cede more responsibilities and privileges to LLMs, it is critical that we’re capable of overseeing and understanding their reasoning, rationales, and behaviors. Therefore, all statistical tests, figures, and conclusions in the paper are framed around relative differences between models’ knowledge transfer capabilities, so a human‑human condition would not change any of the relative rankings that constitute our core quantitative contributions. Additionally, we do not make any claims as to whether the interactions observed in Section 6 apply only to Human-AI collaboration: they could be (and most likely are) general collaboration paradigms that could also apply to Human-Human collaboration. Thus, not having a human-human condition does not discount any of our findings. We do acknowledge that providing a comparison to Human-Human collaboration mechanisms could be a very interesting auxiliary result that links our work to existing work in Human-Human collaboration. We are excited to explore this in the future.

2. Generic collaboration benefits are controlled for implicitly

Any generic boost from working with a partner, be it human or AI, acts as a constant offset that applies equally to every LLM condition. Because participants were randomly assigned to one of eight models, these generic effects are evenly distributed and cannot explain the systematic, model‑specific differences in our findings. In fact, our mixed‑effects logistic regression (Sec. 5, App. C.3) shows that model identity remains a highly significant predictor (p < 0.001) of post‑collaboration success even after controlling for user skill, task type, interaction length, and LLM familiarity. This suggests that the effects we report are tied to which AI collaborator is present, not merely to collaboration per se.

3. Methodological and logistical considerations

Including a human-human baseline would likely also have required significantly more resources and demands on our human participants than we had available, diverting from the core model focused factors we aimed to study.

To your point, we have revised our paper to ensure this point is clear, including clarifying in Section 4.1 why our hypotheses and analyses do not rely on such a baseline, as well as guiding readers to related literature in the related work that reports human‑human transfer rates.

Rationale on prohibiting note taking and pseudocode, and prohibiting participants from accessing the AI during the implementation phase

The central outcome variable in our study is knowledge that participants have actually internalized after collaborating with an LLM. Allowing external memory aids (hand‑written notes, screenshots of pseudocode, or live re‑queries to the model) would blur that signal by letting participants outsource recall rather than encode and retrieve the material themselves. In pilot sessions we observed exactly this behaviour: if participants could jot down or copy‑paste an answer scaffold, many stopped asking clarifying questions and could perform on the post‑task even when they clearly had not grasped the underlying concept.

The manipulation follows the same logic as a closed‑book exam in educational research: retrieval practice without aids is a stronger indicator of true learning than recognition with aids. Classic work on the testing effect [1] shows that removing external prompts yields better long‑term retention and cleaner measurement of what was learned. Our prohibition of pseudocode plays a parallel role in math tasks, where a line‑by‑line algorithm can act as a de‑facto crib sheet.

Note that these restrictions were uniform for all eight model conditions. Therefore they also cannot explain the model‑specific performance differences we report; at worst they lower absolute scores equally, leaving relative ordering intact. To your point, we have added a note in the limitations section acknowledging that real‑world programming often is “open‑book.” A useful next step is to replicate our framework in an open‑resource setting and compare retention under different aid regimes. We believe that our current closed‑book design, however, is the necessary first step to quantify transfer in isolation.

What varies between the three zero-shot attempts for measuring model skill?

Because at test time during collaboration we sample at non-zero temperature, therefore, we only provide humans that the model repeatedly gets wrong to reduce variance and to ensure that at test time, collaboration is meaningful. This creates the need to sample multiple times to determine this.

Why the different submission attempt limits (10 for code, 5 for math)?

This is because the space of possible answers for code (imagine a code function) is much larger than math, which is often confined to integers from 1-1000 for AIME questions. Empirically during our initial testing we found that giving too many attempts for math allowed for a small, but non-zero chance of completely guessing the answer correctly. Additionally, for math, we observed that participants tended to not utilize additional attempts past ~5 attempts effectively (i.e. random guessing behavior arose). These numbers were determined through post-user surveys + manually inspecting solve sessions in a testing phase before the actual experiment. Our rationale for design decisions based on our trial experiments were not added to the paper and this was definitely an oversight: we have now added this to appendix section C. We really appreciate your pointing out details that you find unclear!

It is more likely a demonstration of a ceiling effect, where models with high initial solo performance have an inherently smaller margin for improvement, rather than a failure of the collaboration itself.

We agree that there could be ceiling effect: however, we also observe that Claude-3.7-sonnet is able to exceed Gemini’s ability in inducing collaborative performance in math and closely matching it in code, despite only having half and a third of Gemini's solo performance, respectively (Section 5). Our description of Gemini’s reduced collaborative efficacy thus more points to the fact that despite its high performance, it was not able to output a proportionally higher performance in humans. We don’t discount the possibility of a ceiling effect: but it’s hard to say either way without more data points with higher model performance, which was not possible at the time as Gemini represented the highest performance in our tested benchmark and there were simply no better models then. Nonetheless we think this is a great nuance to include, and we have added this in our discussion of results in Section 5.

For example, was the preference for formal notation in math observed even when users explicitly asked for simpler methods? Could this effect be hampered by the protocol's prohibition of pseudocode, which is often a much more accessible form of communication?

As discussed previously, we must prohibit pseudocode to ensure our experimental procedure accurate measures internalized knowledge. We find that this constraint does not force models into dense symbolic exposition; they were free to, and often did, answer in ordinary prose. In our logs we see many exchanges where participants explicitly requested “plain English” or “a simpler explanation,” and the LLM immediately shifted style, because the guard‑prompt only disallowed verbatim algorithmic scaffolds, not natural‑language reasoning. Hence the occasional preference for formal notation arises from model defaults and user choices, not from any protocol.

Should participants know which model they're paired with, and if not, were they able to infer the model's identity?

Participants were not informed which model they were paired with. We also did not observe any attempts by participants to identify the model through prompting or other means.

The choice of uniform temperature of 0.7 requires justification, especially since the assertion that this reflects how users "commonly interact" is debatable…

For sure. We fixed the temperature to eliminate temperature as a confounder when comparing knowledge‑transfer effectiveness. We chose 0.7 because it is commonly recommended by providers (ie OpenAI in their API docs), balances diversity and determinism, and unlike model‑specific web presets, can be applied uniformly through the API. Because we lack visibility into the proprietary system prompts that shape web‑interface behaviour, controlling this parameter ensures a level playing field and keeps observed differences attributable to the models themselves rather than to decoding settings.

Hopefully this additional context resolves your concerns. Thank you genuinely for your detailed review: reading through your review was invaluable to helping us improve our paper based upon your suggestions. We have already made many clarifying additions in response to your feedback. We believe strongly in our work, so we look forward to continuing the conversation to improve your perception of our work and clear up any additional questions!

References:

[1] Roediger, H. L. III, & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255.

Thank you for this very thorough and thoughtful review. We are very happy that you recognize the significance and the high potential of our work. In summary, we believe that the main issues you have with our work can be fully addressed by providing clarification on the “fixed elo” aspect of our question assignment process.

Addressing Core Issues:

Therefore: the problem is not that it seeks to answer these questions, but that the design is framed to capture one and the results are analyzed to answer the other.

Figure 1 (left)... I do not understand how one could draw this conclusion from the figure.

How can you be sure that it is the AI that enables good performance in human-AI collaborations?

We completely understand your concern, but we do not believe this to be the case. Our experiment is designed so that every participant tackles problems they have only a constant, well‑quantified chance of solving unaided. During each session we select a task whose difficulty is, on average, a fixed amount above the participant’s own ELO: specifically, code $+300 \pm 100$ Elo, math $+1 \pm 0.25$ Elo.  Because task and participant Elos are calibrated with a Bradley-Terry model, the Elo difference immediately gives an estimated solo‑success probability via the logistic Bradley–Terry curve. Empirically, this yields expected unaided accuracies of  ~14 % for code and ~8 % for math. Because the solo‑success probability (on average) is a fixed, known constant for every participant, subtracting it would just shift every point on Figure 1 downward by the same amount and leave all the relative distances intact. In other words, the collaboration accuracy already tells us how much ground the Human + AI pair gains over that baseline. For instance, if the team achieves 50 % accuracy in code, the model is delivering an extra $50-14=36$% of performance, exactly the knowledge transfer we care about. So an explicit “collaboration minus baseline” plot would deliver similar conclusions.

To be certain that this calibration was valid (i.e., that participants truly felt the tasks were just out of reach), we ran a preliminary cohort in which we asked, post‑interaction, whether they believed they could have solved the task unaided. The answers supported our Elo gaps, and we locked them in before running the full study. We also ensured balanced skill distributions across treatment groups (details in Section 4.2/4.3 and Appendix B), so any remaining variance comes from the models, not from uneven human ability. In short, the fixed‑ELO protocol lets us treat collaboration accuracy as a quantitative proxy for knowledge transferred from model to human. This perspective was very helpful, and we have updated Section 4 to make this a lot clearer in our paper by fully detailing this design decision.

What I see instead is that: 1) for some models, such as GPT-4o both math and coding, the human AI collaboration has much higher performance than the models, which indicates that human ability is the source of the collaboration performance, 2) for some models, such as Gem-2.5, the value of collaboration is negative as humans are not able to learn from LLMs

Hopefully this is also addressed in the first response!

Come to think of it, how could one ensure that participants did not just remember the solution that was proposed by the LLM?

This is a great point, and is explained in Section 3.1. Essentially, we purposely choose automatically verifiable domains where there is an “ideation” phase and an “implementation” phase. Much prior work in cognitive transfer and learning has shown that transforming conceptual instructions into executable solutions requires far more than rote memory [4, 5]. To describe this very concretely, take this canonical solution of a sample coding problem provided by GPT-4o:

To solve this problem, we simulate the process of dropping each square onto the X-axis, keeping track of where each square lands and updating the current maximum stack height accordingly. Strategy: Iterate through the list of squares.

For each square, check all previously dropped squares to see if they overlap horizontally.

If they do, take the maximum height from those squares as the base height.

The new height is then this base height + side length of the current square.

Track the maximum height so far after each drop

Even if participants were to memorize the model’s explanations verbatim, successfully implementing these ideas in code demands a deeper understanding. Translating natural language descriptions into working solutions requires selecting appropriate data structures, correctly implementing the algorithm, and efficiently structuring code: all of which go beyond rote recall. This is precisely why we explicitly disallow direct code generation (or symbolic computation in the case of math) as stated in section 4.1; such outputs could be reused without comprehension, undermining the goal of evaluating genuine knowledge transfer. Moreover, the act of debugging (in programming) or diagnosing incorrect reasoning (in math) serves as an additional filter for understanding. Without grasping the intent and mechanics of a proposed solution, participants cannot meaningfully adapt or correct it.

I think this paper would gain from clearly specifying that the research is limited to tasks and workflows in which humans maintain the primary agency, as is mentioned briefly on line 110.

Absolutely. Although we do mention it, we will make this clearer as a limitation in the final version.

The section on results lacks the most important result, namely a comparison between Human performance and Collaborative performance.

In a similar vein to the first response, this information is fully encapsulated in the model performance v collaboration performance comparison.

it would surprise me if someone in pedagogy or learning sciences have not tried to quantify the transfer of learning between humans.

This is a wonderful point. After doing some further review, we were able to find some additional related papers [1, 2, 3], and we greatly appreciate this pointer. We have added this to the revision of the paper!

Addressing Minor Issues:

How did you end up with the specific thresholds (+/- 220 ELO score for code problems and +/- 0.25 for math problems)?

We calibrated this increase on a test cohort. As mentioned in the paper in Section 4.2, we wanted to ensure that these problems represented problems participants could not solve alone, but still had some potential of solving with assistance. This is a detail that we have expanded upon in the paper, as it provides a lot of context.

On line 181 it is stated that subjective human preferences make it hard to evaluating human-AI collaboration. How come?

This can be clearer for sure. What we mean is that a generalized metric for human-AI collaboration is difficult since it would need to adequately factor in subjective preferences and objective helpfulness. Ie, a very helpful teacher that forces you to do practice problems before solving the core problem may not necessarily be liked.

Table 1: What does HTM, HMT, and MHT mean? Please explain.

Thank you for bringing this up. This was a poor design decision we made due to space constraints. Essentially it refers to the relative ELO ratings of model, task, and human. (HTM = Human < Task < Model, HMT = Human < Model < Task… etc.).

Regarding minimal impact of LLM familiarity. I am not sure I follow your hypothesis. Could it be that the users did not actually learn much from collaboration? Have you controlled for this?

This means that our participants’ self-reported prior experience with LLMs did not impact the collaboration performance in a statistically significant fashion. We controlled and tested for potential confounders and covariates as much as possible (detailed in Section 5), and one possible confounder we thought of was whether a human with more experience prompting and interacting with LLMs might give them an edge in solving more problems, no matter which model they were interacting with. Our results show that this was not the case: which is interesting in itself!

Thank you again for your thoroughly detailed review: reading through your understanding of the paper is invaluable to helping us understand what is not clear from a new perspective: especially your thoughts on Figure 1. We have already made many edits in response to your concerns. We believe strongly in our work, so we look forward to further conversation to refine our work and clear up any additional questions!

References:

[1] S. Hajian, “Transfer of Learning and Teaching: A Review of Transfer Theories and Effective Instructional Practices,” IAFOR J. Educ., vol. 7, no. 1, pp. 93–111, 2019.

[2] K. Watanabe et al., “Accelerating Knowledge Transfer by Sensing and Actuating Social‑Cognitive States,” in Proc. UbiComp/ISWC ’23 Adjunct, 2023, pp. 258–262.

[3] O. Elnaggar and R. Arelhi, “Quantification of Knowledge Exchange Within Classrooms: An AI‑Based Approach,” in Eur. Conf. Educ. 2021 Proc., 2021, pp. 221–231.

[4] K. R. Koedinger and I. Roll, “Learning to Think: Cognitive Mechanisms of Knowledge Transfer,” in The Oxford Handbook of Thinking and Reasoning, K. J. Holyoak and R. G. Morrison, Eds. Oxford: Oxford Univ. Press, 2012, pp. 699–715.

[5] O. Chen, E. Retnowati, B. K. Y. Chan, and S. Kalyuga, “The Effect of Worked Examples on Learning Solution Steps and Knowledge Transfer,” Educ. Psychol., vol. 43, no. 8, pp. 914–928, 2023.