Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space

Dear reviewer o8es,

Thank you so much for thoroughly understanding, and positively evaluating our work. We are glad to hear that you enjoyed our work, and found all of our three main findings to be “quite novel and could provide many insights to the community”. We thank you also for your very insightful and actionable suggestions to robustify our claims. In response to your thoughtful suggestions, we addressed your concerns by adding:
(i) a new schematic diagram inspired by [2] Ren, Yi et al.
(ii) further experiments in a more complex scenario with three concept variables yielding Fig. R1 and R3
(iii) a better grounding into the learning dynamics of compositional generalization(CG) literature [1,2]
(iv) a general improvement of figure qualities.

---

• 1. Added New Diagram for the framework: Inspired by [2] Ren, Yi et al., we have updated our framework section of the paper to be easier to understand. We have added a schematic Figure describing the relation of $\mathcal{S}$, $\mathcal{G}$, $\mathcal{M}$, $\mathcal{X}$ in appendix as well. We haven’t been able to fit this new figure in the attached pdf due to space constraints, but this new figure will be included in the final version of this submission.

• 2. Concept space findings generalize to more complex scenarios: We have now extended our results to more complex scenarios. In Fig. R1, we show that our findings on hidden emergence and also our probing methodology generalizes to a compositional setup on CelebA. In this experiment, the model did not show the CG behavior, while latent linear intervention clearly demonstrated the emergence of the capability. We have also generalized our findings on concept signals in Section 4.1 to a 2x2x2 setup with 3 concept variables. We again find that concept signal controls the speed of learning in scenarios involving more than 2 concept variables.

• 3. Practical Implications of Early Capabilities: One implication of the hidden emergence of capabilities is that standard evaluation pipelines on a fixed dataset might be probing behavior, which we show is very seed dependent. Additionally, this motivates studies on enhancing current models by understanding their internals as it suggests that models can have capabilities which are not elicited unless intervened appropriately. With your specific point regarding early stopping, we believe that these models do not explicitly over-fit and nor do frontier diffusion models (see [3]), as they are in the under-parametrized regime and usually trained in an online setup. We also do not have evidence of any capabilities getting lost as we train far beyond fitting the training set.

---

Questions:

• Figure 2-bc legend: Thank you for this feedback! We have made significant revisions to explain the axes better in the captions and to improve the overall quality of the figure.

• Zigzag Pattern: Thank you for this reference [1]! Indeed, our current understanding is similar to [1], where higher difficulty corresponds to lower concept signal. In our case a concept is harder when its concept signal is less important to reduce the diffusion loss. We will make sure to cite [1] and discuss the similarity of the mechanisms underlying the shared trend in learning dynamics.

• Usage of $\sigma$: Thank you for this suggestion! We updated the sigmoid function to $g$ to avoid confusion.

• Clarification of line 216: Yes! We will clarify this in our updated manuscript.

• Figure 6 Legend Thank you for this catch, we will edit this.

• Controlling the phase transition: We did find out that many hyperparameters control the transition, such as weight decay, weight initialization, optimizer, etc.. However, we did not yet find a clear identifiable relationship which allows any predictive hyperparameter transfer. Testing the generalization of these observations to more complex text-to-image models is a great future direction. Currently, we are interested in following up with a theoretical model based on the concept signal values.

• Relation to reference [2]: Thank you for pointing out this work. We will definitely cite and discuss it in the final version! The simplicity bias explanation is very interesting, but it seems non-trivial to define it in our setup at the current stage. We briefly looked into weight decay to add an explicit “simplicity” enforcing term, however this did not qualitatively change our results.

---

Summary: Thank you again for carefully going through our manuscript and giving us such a concrete and insightful list for improvements! As clarified above, your suggestions inspired us to not only create new schematic figures, but also to run 8 more experiments resulting in 2 new experimental plots. With the above, we hope that we have fully addressed your concerns and now you could strongly recommend acceptance of this paper.

---

[3] Kadkhodaie, Zahra, et al. "Generalization in diffusion models arises from geometry-adaptive harmonic representation." arXiv preprint arXiv:2310.02557 (2023).

Dear reviewer uzj6,

We thank you for your detailed feedback! We are glad you found our concept space framework and the robust emergence of capability insightful and interesting. We are especially happy to hear that our work provides “insights into understanding and controlling the development of AI models' capability and behavior”, as this was one of the main goals of our study. In response to your comments, we have added:
(i) An improved description of our framework in Section 3.
(ii) experiments on frontier multimodal embedding (CLIP) and generative (Stable Diffusion v2.1) models yielding Fig. R2.

---

• Concept Signal can be altered by color differences: The concept signal $\sigma_i$ is the absolute change of the data generated when the corresponding concept variable is altered. Here, we are changing the raw RGB difference between the color concept blue and red. Thus ,a rescaling of this difference by $s$ is equivalent to the change of the data generating process $\mathcal{G}\to\mathcal{G}'$, where $|\frac{\partial\mathcal{G}'}{\partial`color`}|=s|\frac{\partial\mathcal{G}}{\partial`color`}|$, and thus results in an enhancement of the color concept signal by a factor of $s$. For a visual explanation, please see Fig.7, where the distances are not schematic, but calculated from the dataset as we have access to the data generating process. We have updated Section 3, where we introduce our framework, so that this relation is more clear. We have also moved Fig.7 to the main text. Thank you for prompting this important clarification!

• Derivation of Equation 1: Thank you for your question. Equation (1) is a phenomenological model designed to qualitatively reproduce concept learning dynamics, where our goal was to suggest a novel potential space for future theoretical study. The sigmoid function models sudden capability acquisition, while quadratic terms represent the energy landscape guiding the model towards correct concept values. Although not derived from a specific learning process, it introduces a new perspective of defining "concept variables" and describing their evolution via differential equations. We are currently working on a theoretical follow-up to rigorously derive this equation from a concrete model that further substantiates our hypothesis. We will clarify our motivation and limitations in an updated draft.

• Your feedbacks have inspired us to validate how many of our observations generalize to real-world data scenarios!: Thank you so much for your insightful suggestions to further explore whether our observations validated in our synthetic setup generalize to real-world data. While, we've made the "simplicity-interpretability" trade-off to derive a set of hypothesis, you are right that it's very important to then go back to realistic scenarios to further validate those hypotheses.

1. Latent linear intervention elicits hidden capabilities also in real world facial attribute dataset (CelebA). [Fig R1] In Fig R1a, we first show how diffusion model struggles to compose unseen combination of "female" and "with hat" concepts with naive prompting. Then, in Fig R1b, we demonstrate how applying latent linear intervention elicits this capability as was seen earlier in synthetic dataset!
2. Square/cubic structure of concept graphs can be found in CLIP embeddings. We show that the concept space framework is useful to understand the failures and success of CG. In some cases, the concept space can be constructed for real diffusion models since it only requires a feature detector (e.g. CLIP), which is much simpler compared to training a generative model. In Fig R2a, we show that CLIP is ready to serve as a feature detector to construct an orthogonal concept space.
3. One specific insight from overprompting and latent linear intervention is that one could use model interventions to evaluate model capabilities. In Fig. R2b, we show an explicit example where overprompting results in elicitation of a CG capability. We speculate that this could be scaled, as we show in Fig. R2a that CLIP can already serve as an orthogonal feature detector, and the lack of an expected feature can be used to provide feedback onto a method guiding generation by intervention.
4. Our work surfaces that controlling concept learning might be more difficult than expected. Our results indicate that even In a synthetic setup where the acquisition of capabilities is robust, the model's apparent ability to CG might appear limited, and can show high variance across run seeds. Our contribution can be understood as attributing the high uncertainties observed in CG in real models to the variance of behavior seen in models with equivalent capabilities.

---

• Summary: Thank you again for your review. Your comments have motivated us to enhance the introduction of our framework in the manuscript and to demonstrate scalability to large models and practical setups. As a result, we have managed to show that our intervention strategies could be valid in Stable Diffusion v2.1 and that CLIP embeddings already encompass orthogonal concepts graphs, suggesting a promising direction for scalability. We hope your concerns have been addressed with these clarifications and experiments, and we would be very glad if you could recommend our paper more strongly.

Dear reviewer G81k,

We thank the reviewer for their very detailed feedback. We are happy you found our findings in 4.4 “surprising and notable”, just the way we felt, and 5. and 4.2 “helpful” and “useful”. In response to your comments and concerns, we have added:
(i) A clear definition and clarification of ``Capability” in Sec. 3.
(ii) A demonstration that turns in concept space corresponds to emergence of capabilities, yielding Fig. R5 b,c
(iii) A substantially more thorough related work section.
(iv) A better quantification of uncertainties, yielding Fig. R4 and R5

---

• 1. Clarification of Capability in 4.1-4.3 vs 4.4: The model’s capability is only discussed in 4.4. We do not mention capability in 4.1~4.3 as we are simply probing its behavior/execution of the task. Nevertheless, we now see that this can be confusing so we clearly defined capability and made this distinction in Sec. 3.
  Your comment also suggested that the relationship between concept space and capability should be strengthened. We have now added Fig. R5 b,c, which respectively demonstrates that 1) sudden turns in concept space does correspond to emergence of capability to compositionally generalize (CG) and 2) this correspondence is robust across different concept signal levels. We hope this clarifies the relation of section 4.1~4.3(concept space) and 4.4(hidden emergence).

• 2. Motivation of the phenomenological model was to postulate concept learning dynamics hypothesis: Eq. (1) is a phenomenological model designed to qualitatively reproduce learning dynamics, requiring different values of c1 and c2 to specify the model generalization point. The initial tendency towards the correlated training point is a result of the sigmoidal function definition, illustrating how different concepts evolve over time. Although the model is not derived from a specific learning process, it introduces the idea of defining "concept variables" and describing their evolution with differential equations. We are currently working on a theoretical follow-up to rigorously derive this equation and further substantiate our hypothesis.

• 3. A more thorough related work section: Thank you for the feedback! We have significantly expanded the related work section now, including discussion of prior work on CG, concept learning, use of interpretability tools to understand learning dynamics, and distinction between capabilities and behaviors. Related to CG, we briefly note that prior work has primarily focused on impossibility results, i.e., showing whether neural networks can express compositional solutions (e.g. [1]). To our knowledge, there is only one work focusing on learning dynamics of CG in a generative model [2], and does not include any intervention experiments or underspecified setups. Underspecification's influence on CG has been partially explored by a few papers on disentangled representation learning [3], but again these papers focus on possibility results and not learning dynamics---the target of our study. A position paper by Hutchinson et al. [4] mentions challenges in CG due to underspecification, but has a different goal compared to our work.

• 4. Errors quantified and visualized: Please see Fig. R4,5 for different quantification of errors. In R4, we show that the std. of behavior across seeds is high while it remains very low for probes of capability. In R5a, we visualized the standard error of mean on trajectories in Fig. 2c.

• 5. Clarifications: The intervention methods are more meant to investigate the model's internal capabilities and show that they emerge consistently before behavior. Although they do improve compositional generalization, that is more of a by product rather than a practical method we introduce.

---

Questions:

• 1, 2, 4: We hope the replies above address these questions.
• 3: a) We have now added example images on top of Figure 4. The generated colors depends on the exact intervention we apply, as we would expect for a steerable model.
  b,c) We have now repeated the Linear Latent Intervention and Embedder Patching for all 5 seeds. We found LLI to work perfectly on all seeds and EP on 2 seeds. We don't think these affect our findings as one intervention method is sufficient to demonstrate model capabilities. Embedder Patching is a more subtle technique (higher bar) since it requires the model to have minimal Embedder-CNN compensating weight changes late in training.
  d) This is indeed how we have plotted Figs. 4 a, b! If the question was directed towards Figs. 4 c, d, we note we have now added all individual seeds to the plots, as mentioned above.

---

Minor Notes

• L.29,69,117-118: Thank you for these catches.
• L.32: By “model experimental systems approach”, we mean a small synthetic setup in which we can control the data distribution and probe the resulting model's behavior, allowing one to study questions that are not feasible to do so in real data.
• L.96,99: Yes
• Figure 2: The two gray trajectories are taken from the training data curves to help visualize concept memorization.
• L. 247: What does it mean to mask the token? Set it to zero? Yes, the corresponding element is set to zero.

---

• Summary: Enormous thanks to the reviewer for pointing out very important points to make our submission more persuasive. We believe our work became more clear and robust thanks to the suggestions. Your comments motivated us to substantially strengthen our theory and related work section. It has also motivated us to quantify uncertainties on our experiments yielding Fig. R4, R5, which are essential to join concept space and hidden emergence, the two big pillars of our work. We believe our manuscript greatly improved in quality, and would be very happy if you could recommend our work more strongly.

---

[1] https://arxiv.org/abs/2310.05327

[2] https://arxiv.org/abs/2310.09336

[3] https://arxiv.org/abs/2006.07886

[4] https://arxiv.org/abs/2210.05815

Dear reviewer cQgr,

We thank the reviewer for their detailed feedback. We are excited that you found our work provides a novel lens for analyzing learning dynamics of diffusion models, yielding claims that are interesting, clear, and well-backed. In response, we have added:
(i) A major experiment on CelebA generalizing our findings to more realistic data, Fig. R1
(ii) A test on Stable Diffusion v2.1(SD), Fig R2b
(iii) Experiments on a more complex 2x2x2 setup, Fig. R3

---

• 1. Simplicity of the setup: Please note that our goal in this work was to understand how a diffusion model learns various concepts underlying the data-generating process and learns to compose them. The simple setup allowed us to perform a more precise analysis of the model’s learning dynamics and develop novel hypotheses on concept learning. However, our results on CelebA (R1) shows that this simplicity did not bottleneck our claims: our results do transfer to more realistic settings! Orthogonally, prior work on compositional generalization (CG) often uses similar settings (e.g., toy shapes, colors), making our chosen setup a natural starting point.

• 2. Concept Signal relies on G: Computing precise concept signal values is indeed non-trivial in real data. One approach would be to approximate the derivative of G by training a VAE to compute $dG/dz_i$. However, our primary contribution is a step towards understanding what controls CG when G is known. At the same time, our results show that we can retrospectively establish orders of concept signals (CS) for real data: e.g. we can infer CS for gender is stronger than hat from the CelebA experiment (R1).

• 3. The model for learning dynamics: We clarify that Fig. 2 shows accuracies over training time, where x and y represents accuracy for color and size. The key insight is that concept signals determine the learning dynamics' trajectory. Fig. 2(c) shows that curves bend toward specific directions reflecting the relative strengths of the concept signals for size and color. To model this behavior, we have defined conditions in Eq. 1: $\sigma_1 > \sigma_2$ and $\sigma_2 < \sigma_1$​. We will provide further clarification on these points in the revised version.

• 4. The definition of capability: Thank you for the suggestion! You correctly inferred our intended meaning: by capability, we mean “the model's internal ability to understand and compose concepts”. We have now formalized this in Sec. 3 of the paper.

• 5. Figures: We apologize for the pixelated plots. We updated all figures with higher quality versions. We also labeled all axes and added generated images to aid understanding, showing how the model improves during training (e.g., see R4). As per your suggestions, we have clearly clarified the axes in the caption and labeled the colorbar in Fig. 3. We updated Fig. 4 so that the curves for each seed are clearly visible. We updated Fig. 5b and labeled the color axis as $a$. We thank the reviewer again, as we really value high quality plots especially for a paper focused on sending a qualitative message.

• 6. Linear Latent Intervention(LLI) Breaking Down
  The focus of our interventions is mainly to gain understanding of how capabilities are acquired in a model rather than to present a method to improve CG in practical settings. This is also why we presented 3 methods, so that we do not overfit to pitfalls of one specific method. Thus, our methods are sufficient to show a model has CG capabilities, but not necessary or efficient. Nevertheless, for LLI, we show in R1b that it can generalize to more realistic CelebA data.

• 7. Discussion of Celeb A in main text
  Thank you for the suggestion! We moved some of the CelebA results to the main text.

---

Questions:

• On overprompting(OP): The short answer is “sometimes" and just as LLI above, this intervention method was sufficient to show capability. While there will be cases where OP will not elicit CG, in R2b, we show that OP elicits CG even in SD, demonstrating its generalizability. In fact, SD’s prompt encoder takes brackets [] around words which need to be enhanced, and internally scales the vectors corresponding to these tokens, implementing OP.

• Assumptions on Embedder Patching: Yes, and also an even stronger one: weights in MLP and CNNs should not drift after convergence of training. Restating the point made in 6 above, our intervention methods are sufficient but not necessary to show a model’s capability.

• CelebA Results: Certainly! We reproduce both concept memorization(R1a) and hidden emergence(R1b) in CelebA. The former checks out the inverse concept signal learning point made above and the latter shows that this synthetic setup captures realistic concept learning well.

---

Limitations:

• Comparisons: To our knowledge, the listed tools (e.g., probing) are not standard protocols studying learning dynamics in gen. models. The closest work to ours is Okawa et al. 2023, where the authors used probing to model learning dynamics, but could not identify what determines the order of concept learning, that there is concept memorization, and that there is a hidden emergence of capabilities. Nonetheless, we will add an expanded discussion of these tools in the paper.

• Generalizability: We have added 2x2x2 (R3) and CelebA (R1) results to demonstrate that our result, at least, extends to systems which are a little bit more complex.

---

• Summary: We thank the reviewer again for detailed feedback. The reviewer’s comments motivated us to investigate further into the more realistic CelebA setup, and we are happy to show that our major finding on hidden emergence is well reproduced. We hope our changes and new experiments address the reviewer’s concerns adequately, and would be glad if our paper can be recommended more strongly.

---

[1] arxiv.org/abs/2311.03658