Nonparametric Identification of Latent Concepts

We are genuinely grateful for your insightful comments. In light of these, we have introduced new experiments (https://anonymous.4open.science/r/0-518C/rebuttal_new_results.pdf) and new discussions. Please find our detailed response below:

Q1: Strong assumptions / limited generative process space.

A1: Thank you so much for the comment. It seems we may not be fully aligned on the core setting and assumptions. We sincerely appreciate the opportunity to clarify the following:

• Q1.1: There is a very strong assumption that the generative process $f$ is bijective.

  A1.1: As defined in L129-136 left, $f$ is an unknown diffeomorphism onto its image—a very standard assumption in identifiability literature (e.g., most works mentioned in the survey [Hyvärinen et al. 2024]). It aligns with the manifold hypothesis, which posits that high-dimensional data (e.g., pixels) lie on a lower-dimensional latent manifold (e.g., concepts). In overcomplete settings (latent dim > observed dim), much stronger assumptions (e.g., linearity) are typically required.

  • Hyvärinen et al., Identifiability of latent-variable and structural-equation models: from linear to nonlinear, 2024

• Q1.2: $f$ may be linear (if it denotes $M$); unclear spaces that $f$ maps between.

  A1.2: Thanks for raising this. To clarify, $f$ and $M$ are unrelated.

  • As defined in L113–121 left, $f$ maps latent concepts $\mathbf{z}$ to observations $\mathbf{x}$ (Eq. 1: $\mathbf{x} = f(\mathbf{z})$) and is a general diffeomorphism, without any linearity constraint.

  • As defined in L159 left–133 right, $M$ is a binary structural mask between classes $\mathbf{c}$ and concepts $\mathbf{z}$, independent of $f$.

• Q1.3: All concepts are binary.

  A1.3: As defined in L116 left, all concepts are continuous variables ($\mathbf{z} \subseteq \mathbb{R}^{n}$), with no restriction to binary values.

• Q1.4: Unclear how strong the structural diversity is.

  A1.4: Thank you for the great question. In a general nonparametric setting (no functional/distributional constraints), identifiability is known to be ill-posed without additional assumptions. Structural diversity characterizes when every concept is identifiable in such settings. It is expected that ambiguous cases must be excluded, as in most prior identifiability theories.

  A key contribution of our work is going beyond this assumption. The local comparison framework (Thm. 1, Prop. 1) provides identifiability guarantees even when global condition is violated—allowing recovery of many concepts as groups (e.g., in FFHQ). In contrast, most prior work cannot provide any guarantees under any degree of violation.

  In light of your question, we have also added new experiments under partial violations (Figs. 1–2 in the link), confirming empirical robustness.

Q2: Additional metrics for synthetic and real-world datasets.

A2: Great suggestion. In light of it, we have added evaluations on DCI and Mutual Information Gap (MIG) for synthetic datasets (Figs. 1–2 in the link), and Fréchet Inception Distance (FID) and Perceptual Path Length (PPL) for real-world images (Table 1 in the link). Our method consistently outperforms baselines.

Q3: More baselines.

A3: Thanks for your advice. We have added two more baselines (Table 1 and Figs. 7-8 in the link), further supporting the effectiveness.

Q4: Code is not provided.

A4: Thanks. Due to rebuttal policy, we can only include figures in the link. Based on the flow-based method (L1372), we added an ℓ₁ norm (L369 right) since ℓ₀ norm is known to be indifferentiable.

Q5: Test on CUB or AwA2.

A5: Thank you for the advice. We have conducted new experiments on both datasets (Figs. 5–8, Table 1 in the link), showing that identifiable concepts remain recoverable. With these additions, we now have seven different real-world datasets to support the practical applicability.

Q6: Comparison with two related papers.

A6: Thanks for the suggestion. Please find the detailed comparison below:

• Brady et al., 2023:

  (This work has been discussed in the introduction (L62-76 left).)

  • Theory: Assumes each observed variable (pixel) maps to only one latent slot.

    • In contrast, we allow arbitrary mixing.

  • Practice: Targets recovering concrete objects as pixel groups, assuming no occlusion or overlap among them.

    • In contrast, we focus on general concepts (concrete or abstract), with no constraints on observed object composition.

• Leemann et al., 2023:

  • Theory: Assumes linear generative process or orthogonal Jacobians.

    • In contrast, we allow general nonlinear generative process, where the identifiability is much harder.

  • Practice: Targets post-hoc explanation assuming access to linear combinations of latent factors.

    • In contrast, we focus on general concept learning from observation in a nonparametric setting.

We deeply appreciate your valuable insights. In light of these, we have carefully added several new discussions and conducted new experiments (https://anonymous.4open.science/r/0-518C/rebuttal_new_results.pdf). Please find our detailed responses below:

Q1: More clarifications on Eq. 5.

A1: Thanks for your suggestion, we have added further clarifications accordingly. $\mathbf{T}$ in Eq. 5 is invertible because the mapping $t$ is invertible. The invertibility of $t$ follows from Eq. 4 (L743-746), independent of $g$. Eq. 5 simply represents the derivatives w.r.t. $\mathbf{c}$, and $\theta$ denotes other factors independent of $\mathbf{c}$.

Q2: Is the comparison based on the same method under violated assumptions?

A2: Yes. The estimation method (previous work + regularization) is unchanged; only the data-generating process is altered to violate structural conditions (L407 left). We have now emphasized this more explicitly in light of your great question.

Q3: The technical relation to previous methods such as iVAE and sparsity approaches.

A3: Thanks a lot for the insightful comment. iVAE (and related methods) require sufficient distributional change (e.g., $2n+1$ environments), while we focus on how classes affect concepts—structure, not magnitude, of change. Our theory is closer in spirit to sparsity-based methods, as both leverage structural relationships. However, there are two key differences:

• We study class–concept structures, while prior work focuses on concept–measurement structures. This relaxes assumptions on mixing function by weak supervision from class labels.

• Our pairwise comparison framework supports identifiability under partial violations, unlike prior work that requires full structural assumptions.

We hope this makes the distinctions clearer and would be more than happy to further refine it based on your feedback.

Q4: Possible missingness in L117 left.

A4: Thanks. We were not certain we understood correctly, so we assumed you suggested moving L124-125 ($p(\mathbf{z}|\mathbf{c}) = p(\mathbf{z}_A|\mathbf{c}) p(\mathbf{z}_B)$, $\mathbf{z}_A \coloneqq g(\mathbf{c}, \theta)$) alongside L117 (Eq. 1). We've made this change, but please let us know if this wasn't your intention.

Q5: The set $A$ and the matrix $M$ should be related explicitly.

A5: Thank you–we now explictly highlight that $A_i$ is the support of $M_{i,\cdot}$ in the updated manuscript.

Q6: Abuse of notation with $\theta$ in Defn. 1.

A6: Addressed—we removed $\theta$ in Defn. 1, keeping only $(g, p_{\mathbf{z}}, M)$, and likewise for $\theta’$. Thanks for raising this.

Q7: More clarification on $\mathbf{T}$ and $\mathrm{T}$.

A7: Thank you for your suggestion. We have added further highlights as follows:

• $\mathbf{T}$: a matrix-valued function in $D_{\mathbf{c}} \hat{g} = \mathbf{T} D_\mathbf{c} g$.

• $\mathrm{T}$: a matrix with the same support as $\mathbf{T}$.

Q8: Rename Prop. 1 to Cor. 1.

A8: Thanks, we have renamed it accordingly.

Q9: L266 left — Do you mean “and which cannot be expressed as”? Do you allow singular distributions?

A9: Thanks for the suggestion. We meant “and which cannot be expressed as” and have revised the wording. The condition excludes singular distributions by requiring the subset $A_{\mathbf{z}} \subseteq \mathcal{Z}$ to have non-zero probability measure.

Q10: L360 right — How exactly are the $\mathbf{c}^{(i)}$ created? What is a multi-hot vector? Is it sufficient to use one-hot vectors?

A10: Thank you for the great suggestions. Accordingly, we have added a detailed explanation in the updated manuscript. Specifically, $\mathbf{c}^{(i)}$ denotes the classes of sample $i$: a one-hot vector if single-labeled, or a multi-hot vector if multi-labeled (e.g., $[0,1,1]$ for two active classes).

Q11: Learning by comparison could be connected closer to the experimental methodology and the literature

A11: Thanks for the helpful advice. We have rewritten L82–99 left for closer grounding:

We address this question by grounding our approach in a fundamental cognitive mechanism: humans learn concepts by contrasting diverse classes of observations. Classic studies have shown that concept formation relies on detecting distinctions across examples—Bruner et al. (1956) emphasized learning through contrasts between exemplars and non-exemplars; Gibson (1963, 1969) proposed differentiation as a basis of perceptual learning in infants; and Gentner & Namy (1999) demonstrated that direct comparison enables children to abstract category-defining features. This principle is further supported by extensive literature across cognitive science, reinforcing that learning by comparison is the key underlying engine.

Experimentally, FFHQ (Figs. 13–15) illustrates the need for local comparisons due to entangled concepts. We also added new experiments on two new datasets (see anonymous link), in light of your great insights.

We are profoundly thankful for your insightful feedback. In light of it, we have included new discussions and conducted new experiments (https://anonymous.4open.science/r/0-518C/rebuttal_new_results.pdf). Please find our point-by-point responses below:

Q1: Thm. 1 in high-dimensional space or under noisy observations.

A1: Thank you for the question. Theoretically, the assumption only requires Jacobians to be linearly independent at some points to ensure meaningful variation—which can be expected from large populations. We have also added new experiments in light of your suggestions:

• High-dimensional space: Beyond the five image datasets used before, we added two more (Figs. 5–8, Table 1 in the link). Since images are naturally high-dimensional, these results across seven datasets support applicability in such settings.

• Noisy observations: We added experiments with noisy images (Figs. 3-4 in the link), showing meaningful concepts remain identifiable.

Q2: Practical applicability of structural diversity.

A2: Thanks for the suggestion. We have added new experiments under partial violations (Figs. 1–2 in the link), showing robustness. Intuitively, if a concept (e.g., “furry”) is absent in some classes, it can be disentangled. If all classes share it, it is unidentifiable under Thm. 2. However, our local comparison framework (Thm. 1, Prop. 1) still ensures identifiability even without global structural diversity.

Q3: Reproducibility details for Jacobian regularization and synthetic data.

A3: In light of your suggestion, we have highlighted details further in the updated version. Due to the rebuttal policy, we can only include figures in the anonymous link. We approximate ℓ₀ regularization with an ℓ₁ norm (L369 right) since ℓ₀ norm is known to be indifferentiable. We use a flow-based estimator (Sorrenson et al., 2020) to recover latent concepts (L365–367 right), though our theory is estimator-agnostic—as long as the observed distribution is matched and regularization applied. For synthetic data, we randomly construct a class-concept structure satisfying our conditions and sample each latent variable conditioned on its class labels.

Q4: Suggestions on empirical evaluations.

A4: Thank you so much. We added the following new experiments accordingly:

• Q4.1: Partial violation of structural diversity.

  A4.1: Evaluated in Figs. 1–2 in the anonymous link; results show robustness.

• Q4.2: Quantitative metric for real-world evaluations.

  A4.2: We added Fréchet Inception Distance (FID) and Perceptual Path Length (PPL). As shown in Table 1 in the link, ours consistently outperforms baselines.

• Q4.3: More metrics for simulation.

  A4.3: We added DCI score and Mutual Information Gap (MIG) (Figs. 1–2 in the link); same conclusion holds.

Q5: Entangled concepts on FFHQ.

A5: Thanks for your question. Learning by local comparisons focuses on the unique part of each class (e.g., a group of dependent concepts), enabling partial identification even when full disentanglement is impossible. FFHQ illustrates this: despite global violations, concept groups remain identifiable (L1460–1467). In addition to the full identifiability (shown in other datasets), this partial identifiability is also one of our main focuses.

Q6: Baseline comparisons lack details.

A6: Thanks. As noted in L407 left, all models share the same setting except the data-generating process. We have highlighted this further in the updated manuscript.

Q7: Some steps (e.g., Eqs. 82–83) lack intuition.

A7: Thank you—we have added more intuition accordingly. For any $i \in \\{1,\ldots,n_A\\}$ and $v \in \\{1,\ldots,n_A\\} \setminus i$, $(\pi(i), v) \notin \mathcal{T}$ (apologies for a typo—$k$ in L1053 should be $v$) implies that all entries in row $\pi(i)$ of matrix $\mathrm{T}$ are zero except possibly at index $i$. Since $\mathrm{T}$ is inveritble, each row must have a non-zero entry, implying $(\pi(i), i) \in \mathcal{T}$.

Q8: Measuring structural diversity in practice.

A8: Thanks for the excellent point. We have added a discussion in the updated version. Structural diversity can be assessed via concept–class mutual information (as you kindly suggested) and domain knowledge (e.g., “sunshine” may be universal, “furry” is class-specific). Even if violated or unsure, our local comparison framework ensures alternative identifiability.

Q9: Compare with Delfosse et al., 2024 on differences in identifiability guarantees.

A9: Thanks for the suggestion. For Delfosse et al. (2024), did you mean “Interpretable concept bottlenecks to align reinforcement learning agents”? We are not sure if we understood correctly, as we did not find identifiability theories in that paper. Please let us know if we missed or misunderstood anything—we would be more than happy to include a discussion accordingly.

We sincerely appreciate your constructive feedback. Accordingly, we have included several new discussions in the updated version. Please see our point-by-point responses below:

Q1: More intuition on Theorem 2 and Figure 4.

A1: Thanks for the suggestion. We have added more discussion in the revised version. Theorem 2 guarantees the identifiability of each class-dependent concept under the structural diversity condition. Figure 4 illustrates this condition: for example, for concept $\mathbf{z}_1$, there exist classes $\mathbf{c}_1$ and $\mathbf{c}_3$ s.t. $\mathbf{z}_1$ is unique to $\mathbf{c}_1$, allowing that concept to be disentangled. Meanwhile, class-independent concepts can also be disentangled as a block by leveraging distributional variety.

Q2: What models did the authors use for the real-world datasets?

A2: Thanks for the question. We used a general incompressible-flow network (GIN) following Sorrenson et al., 2020, with an additional sparsity regularization (L365-370 right). We have further highlighted this in the updated manuscript according to your constructive comments.

Q3: Relation of theoretical claims with previous work, especially in the ICA literature.

A3: Thank you for the insightful comment. We truly appreciate the opportunity to further illustrate how our theoretical results relate to prior work. While Thm. 2 shares ICA’s goal of identifying latent variables, it differs in assumptions, techniques, and scope. We would be very grateful for any further feedback if any part of the following differences is unclear or could be better articulated.

• Objective: While ICA aims to recover all latent variables individually, our focus is on identifying which concepts are reliably recoverable, grounded in the cognitive process of comparison. Our main contribution lies in learning by local comparisons—showing that unique concepts can be disentangled from any class pair. This naturally extends to global identifiability (Thm. 2), but the local view enables more flexible, practical analysis. In contrast, most previous work in ICA cannot provide any guarantee with partial violation of their assumptions for some latent variables.

• Techniques: ICA methods are built on elegant and powerful assumptions—such as auxiliary variables or sparse mixing—that enable strong identifiability results. For example, iVAE needs more than 2n+1 environments with sufficiently varying distributions; sparsity-based methods assume sparse concept–observed mixing structure. Our method avoids these by relying on class–concept structures, offering partial identifiability when global assumptions may not fully apply.

In summary, our theory targets a different question: determining which concepts are identifiable under minimal, localized assumptions. We do not claim generality over ICA, but rather provide complementary insights, especially in settings where global assumptions may be partially violated.

Q4: How to extract the concepts from raw input?

A4: Thank you for the great question, and we have added more details in the updated version. After estimation (detailed in A2), we recover a latent vector from observed data and class labels. Each dimension of this vector corresponds to a concept. To aid interpretation, we rank dimensions by their variances—based on the heuristic that higher-variance components are often more semantically meaningful.

Q5: Rewrite the paragraph in lines 82-92, and add a reference from the cogsci literature.

A5: Thanks so much for the great suggestion. We have rewritten the paragraph as follows and added a line of references.

We address this question by grounding our approach in a fundamental cognitive mechanism: humans learn concepts by contrasting diverse classes of observations. Classic studies have shown that concept formation relies on detecting distinctions across examples—Bruner et al. (1956) emphasized learning through contrasts between exemplars and non-exemplars; Gibson (1963, 1969) proposed differentiation as a basis of perceptual learning in infants; and Gentner & Namy (1999) demonstrated that direct comparison enables children to abstract category-defining features. This principle is further supported by extensive literature across cognitive science, reinforcing that learning by comparison is the key underlying engine.

---

References:

Bruner, J. S., Goodnow, J. J., & Austin, G. A. (1956). A Study of Thinking. Wiley.

Gibson, E. J. (1963). Perceptual learning. Annual Review of Psychology.

Gibson, E. J. (1969). Principles of perceptual learning and development. Appleton-Century-Crofts.

Gentner, D., & Namy, L. L. (1999). Comparison in the development of categories. Cognitive Development.

---

Q6: Move intuitions before the theorems.

A6: Thank you very much for the helpful suggestion. We have revised the layout accordingly, placing the intuitions before the theorems for better flow.