Learning Discrete Latent Models from Discrete Observations

We sincerely appreciate the valuable reviews and constructive comments, which helped improve the quality of this paper. We have carefully reviewed the manuscript as per your suggestions. We have revised the paper accordingly, and below, we provide detailed responses to your comments.

W1: I find that Assumption 1.2 appears to be too general. For instance, in the example illustrated in Figure 1, it's possible that X3 could be a cause of X1.

• *A1: Thank you for raising this concern. Assumption 1.2 is indeed practical and aligns with many real-world applications. For instance, as mentioned in the introduction, it holds in contexts such as survey data in social sciences and topic modeling in natural language processing. Figure 1 provides a concrete example to illustrate this point. While it may appear that $X_3$ and $X_1$ are dependent or causally related, it is reasonable to conclude that such relationship becomes insignificant when the disease is observed. In this case, $X_3$ does not serve as a significant cause of $X_1$.

W2: I'm not an expert in ICA and found myself waiting for an explanation of both ICA and the term domain as I read. It would be helpful for the author to inform readers early on that Appendix A1 contains the necessary background knowledge.

• A2: Thank you for your suggestion. We have clarified the term “domain” in the paper by adding the following explanation: Here, “multi-domain” refers to distinct scenarios or conditions under which the distributions of variables differ.

For ICA, we have added a a short explanation of what ICA is in the main text and a reference to the appendix where ICA is discussed in greater detail, allowing readers with limited background to gain a clearer understanding: For instance, in ICA, the observed data is considered a mixture of independent, unobserved components, and the goal is to "demix" these observations to recover the latent variables, up to some ambiguity (please refer to Appendix A.1.1 for more details).

Q1: For the abstract, providing the full name of ICA could be better.

• A1: Thanks for the advice, we have modified in the paper: In approaches where certain uniqueness of representation is guaranteed, such as nonlinear Independent Component Analysis

Q2: For assumption 1.1, should it be $P(X,Z\mid k)=P(X\mid Z)p(Z\mid k)$?

• A2: Thanks for the correction of this typo. We have corrected it as follows: Across all domains, the binary observed variables $\mathbf{X} = \{X_1, \cdots, X_N\}$ are generated from the binary latent variables $\mathbf{Z} = \{Z_1, \cdots, Z_D\}$ according to the factorization $P(\mathbf{X} \mid \mathbf{u}) = \sum_{l=1}^{2^D}P(\mathbf{X} \mid \mathbf{Z}=\mathbf{z}_l)P(\mathbf{Z} =\mathbf{z}_l \mid \mathbf{u})$.

Q3: For Theorem 2, you said after taking the log on both sides of 2. What is 2? And why do the linear equations have unique solutions if and only if $2^{D-1} \ge D$

• A3: Thank you for pointing this out. The “2” refers to Equation 2—apologies for the confusion. We have clarified this in the paper. After taking the logarithm of the equations, they are transformed into linear equations. The identification of parameters in these linear equations depends solely on the relationship between the number of equations and the number of unknowns. According to Equation 2, the number of equations is $2^{D-1}$, while the number of unknowns is D, as we assume the binary latent variables are independent of each other. Therefore, identifiability is established if and only if $2^{D-1} \geq D$.

Q4: For line 652, please add more examples.

• A4: Thank you for pointing this out. We have updated the paper to include the following revision:: Domain information is widely observed in the real world, such as labels across populations, different experimental conditions, varying time periods, geographic regions, or demographic groups (e.g., age, gender, or socioeconomic status). These distinctions are often integral to many datasets, rendering the theorem adaptable to various downstream problems.

Q5: For line 231-232, why for $\alpha$, the l ranges to $2^D - 1$?

• A5: Thank you for the question. For line 231-232, we assume that the latent variables in the model can be dependent. In that case, the free parameters $\alpha$ for characterizing the marginal distributions of latent variables ($\mathbf{Z}$) in each domain have the total number of $2^D - 1$. For better clarification, we have revised the content as follows: Based on Equation 1, we introduce additional notations to better represent the free parameters we aim to identify. Let $\alpha_{l,k} = P(\mathbf{Z} = \mathbf{z}_l \mid k)$, where $l \in (\{1, \dots, 2^D - 1\} )$, represent the free parameters used to characterize the marginal probabilities of the latent variables.

We sincerely appreciate your insightful and constructive feedback, which has greatly enhanced the clarity of our theories and models, as well as the robustness of our experiments. In response, we have updated the paper accordingly. Below, please find our detailed responses to your comments.

W1: The empirical examples are both not really discrete but rather discretized, which of course is not ideal. Since the paper is primarily theoretical and just has some empirical examples as a “nice addition”, I think this is fine.

• A1: Thank you for pointing this out. For the two empirical examples, the original observed variables from Big Five are discrete, as they measure five levels of agreement. In contrast, the NASDAQ-listed dataset is continuous. Since real-world data can often be inherently discrete or discretized during collection or preprocessing, we manually discretized this dataset to fit the scope of our study. The goal of this section is to demonstrate that regardless of the data’s generating process—whether naturally discrete or discretized—we can reliably recover the latent variable distributions and uncover meaningful relationships between latent and observed variables. This is achieved without imposing strict constraints on the data generation process, demonstrating the robustness of our approach.

W2: Still I would like to see real examples, where we actually have both discrete observed and discrete latent variables. The authors argue with the cases where we would, for example, discretize personality traits, which as a psychologist by training originally, I have to reject as not a good (oversimplifying) approach.

• A2: Thank you for your thoughtful comment and we understand the concern. For the personality traits case, the observed variables are categorical, as they are measured on a scale with five levels of agreement. There are indeed many real-world applications where both observed and latent variables are discrete. For instance, as shown in Figure 1, in medicine, symptoms (observed) and potential diseases (latent) are frequently represented as binary variables. This highlights the practical importance of our approach in scenarios involving discrete data. In social sciences, survey data frequently involve discrete observed and latent variables, such as levels of satisfaction for questions and underlying respondent traits. Another concrete example from social science is the Trends in Mathematics and Science Study (TIMSS), an international, large-scale assessment that evaluates the mathematics skills and science knowledge of students across different grades. The variables in this dataset are binary and are widely analyzed using latent class models.

W3: The authors often state the inequalities involving N, K, D as weak, e.g Assumption 4.1 and similar. Can the authors explain a bit more on why they think these are weak? I.e. can we expect these to be justified in most real world cases?

• A3: Thank you for your question. To clarify, in the inequality constraints, N represents the number of observed variables, K represents the number of domains, and D represents the number of latent variables.

• The assumptions we make are relatively mild, as many datasets typically have a sufficient number of observed variables. However, we acknowledge that the applicability of these conditions depends on the specific data. For instance, if the number of observed variables is limited, achieving the sufficient conditions for identifiability may require a large number of domains (potentially even as many as the observed variables). That said, under alternative settings—such as when the latent variables are assumed to be independent—the conditions may be relaxed.

• We also recognize that there are real-world scenarios where the sufficient conditions may not hold. For example, when there is only one observed variable in just one domain, the model is certainly unidentifiable. Addressing such cases and generalizing our current framework is an important direction for future work, and we are keen to explore this in subsequent research.

Thank you for your insightful and constructive feedback, which indeed improves the readability of our theories and models as well as the soundness of our experiments. We provide the point-to-point response to your comments below and have updated the paper accordingly. Please find our point-by-point responses below.

W1: The introduction of K domains is not justified. What are these domains modeling? Why are they necessary in the analysis?

• A1: Thank you for your thoughtful questions. The multi-domain setup is an inherent property of our model, meaning that the distributions of latent variables depend on an auxiliary variable as part of the model generation process. We utilize this property in establishing the identifiability of the model.

• The motivation for introducing multi-domain settings in our work is as follows: Most, if not all, previous studies on identifiability in multi-domain scenarios assume that the variables are continuous. In contrast, we are the first to establish identifiability in the discrete case and demonstrate how multi-domain information can significantly relax the parameter constraints that would otherwise be more restrictive in single-domain settings.

• For example, assuming independent latent variables, when the number of latent variables is 3 $(D=3)$ and the number of observed variables is also 3 $(N=3)$, the distributions are not locally identifiable, as demonstrated by the simulation results in Table 1. However, if we have 6 or more domains, we can effectively identify all distributions, even when only 3 observed variables are available per domain.

• In real-world applications, obtaining multiple domains is often practical and feasible. For instance, participants’ demographic information in survey data, user information in recommender systems, and object categories in image or video analysis are examples where multi-domain information is readily available and can be leveraged.

W2: Definition 1 is not precise, it needs to be mathematized in order to be relied on to follow proofs;

• A1: Thanks for your suggestion, we have revised that in the paper as follows:
• Let $X$ be a random variable with a discrete set of possible states $\{s_1, s_2, \dots, s_k\}$ and an associated probability distribution $P(X = s_i) = p_i$ for $i = 1, \dots, k$, where $p_i \geq 0$ and $\sum_{i=1}^k p_i = 1$. The random variable $X$ is said to have state-level identifiability if its distribution is identifiable up to a permutation of its states. Formally, this means there exists a permutation $\sigma \in S_k$ such that: $P_Y = [\{p_{\sigma(1)}, p_{\sigma(2)}, \dots, p_{\sigma(k)}\}$], where $\sigma$ is an element of the symmetric group $S_k$, representing all possible permutations of $\{1, 2, \dots, k\}$. In other words, the specific labeling of the states $\{s_1, s_2, \dots, s_k\}$ may not be recoverable, but the probability distribution over the states $\{p_1, p_2, \dots, p_k\}$ is uniquely determined up to a permutation of the states.

W3: The proof of Theorem 3 is unclear. Also, assuming that the proof is correct, given how trivial it is, does this result really deserve the name of "Theorem"?

• A1: Thank you for pointing this out. I assume you are referring to Theorem 2, as the proof of Theorem 3 is quite comprehensive in the appendix. Please feel free to correct me if this is not what you intended. For Theorem 2, we hope you agree that it is more appropriate to use “Lemma” instead of “Theorem” in this context. We have updated the manuscript accordingly.

W4: The definition of local identifiability needs to be (1) mathematized, (2) moved to the main text, as it seems pretty central to the paper's focus;

• A4: Thank you for your suggestion, we have mathematized it and moved it into the main text in the updated manuscript:

• Let $\mathcal{M}(\theta)$ denote a parametric model, where $\theta \in \Theta$ represents the parameter vector in the parameter space $\Theta$, and $\mathcal{M}(\theta)$ maps the parameter $\theta$ to the set of observed data distributions. The model $\mathcal{M}(\theta)$ is locally identifiable at $\theta_0 \in \Theta$ if: $\exists \, \epsilon > 0 \text{ such that } \forall \theta \in B_\epsilon(\theta_0) \cap \Theta, \quad \mathcal{M}(\theta) = \mathcal{M}(\theta_0) \implies \theta = \theta_0,$ where: $B_\epsilon(\theta_0) = \{ \theta \in \Theta : \|\theta - \theta_0\| < \epsilon \}$ is the open ball of radius $\epsilon$ centered at $\theta_0$, $\|\cdot\|$ denotes a norm (e.g., Euclidean norm) on the parameter space $\Theta$, $\mathcal{M}(\theta)$ is the mapping from parameters $\theta$ to the observed data distributions.

Thank you for your insightful and constructive feedback, which indeed improves the readability of our theories and models as well as the soundness of our experiments. We provide the point-to-point response to your comments below and have updated the paper accordingly.

With gratitude, we have included a new section summarizing the conditions for each type of identifiability. We have also added more explanations or details to the assumptions and experiments. The manuscript has been thoroughly revised in light of your constructive suggestions. Please find our point-by-point responses below.

W1: While the section ordering is logical, the paper's writing is confusing, and lacks a well-defined summary of conditions for each type of identifiability. Some assumptions are stated redundantly (e.g., latent factors being mutually independent within each domain, positive free parameters), and some results appear unnecessary, adding to the confusion.

• A1:1 This is a great suggestion, below please see a summary of conditions and a combination of conditions for each type of identifiability. It is also updated in the manuscript.

1. One-to-one mapping + Assumption [1], [2]: strict identifiability on $P(X_i\mid \mathbf{Z})$ and $P(\mathbf{Z}\mid \mathbf{u})$.
2. One-to-one mapping + Assumption [1], [2], [3]: strict identifiability on $P(X_i\mid \mathbf{Z})$ and $P(Z_j \mid \mathbf{u})$.
3. Flexible mapping + Assumption [1], [4]: local identifiability on $P(X_i\mid \mathbf{Z})$ and $P(\mathbf{Z}\mid \mathbf{u})$.
4. Flexible mapping + Assumption [1], [3], [5]: local identifiability on $P(X_i\mid \mathbf{Z})$ and $P(Z_j \mid \mathbf{u})$.
5. Flexible mapping + Assumption [1], [6]: strict identifiability on $P(X_i\mid \mathbf{Z})$ and $P(\mathbf{Z}\mid \mathbf{u})$.
6. Flexible mapping + Assumption [1], [3],[6],[7]: strict identifiability on $P(X_i\mid \mathbf{Z})$ and $P(Z_j \mid \mathbf{u})$.

• A1.2 Thank you for your suggestions regarding the paper’s structure. In our view, it was necessary to state certain assumptions, such as those concerning positive free parameters, separately for different theorems because the constraints apply to distinct parameters across them. To address potential redundancy, we highlighted the differences in the assumptions involved in the various results. For example, in Assumption 5, we replaced "The free parameters are all positive" with: "The free parameters {$\gamma_{(1,1)}, \cdots, \gamma_{(D,K)}, \beta_{(1,1)}, \cdots, \beta_{(N,2^D)}$}are all positive." Hope this can address your concern properly.

W2: The assumptions, which are central to the paper as they constitute the conditions for identifiability, require more explanation and discussion of their practicality:

W2.1: Assumption 1.2 is unrealistic in medicine, for example (although admittedly common in the field)

• A2.1: We appreciate your observation and acknowledge that, in practice, observed variables may sometimes influence each other. Allowing directed interactions among measured variables would be a very interesting line of future research. In this paper, we primarily focus on scenarios where observed variables are conditionally independent given the latent variables—a setting that applies to many practical situations. For instance, in healthcare, as shown in Figure 1, symptoms are often conditionally independent when conditioned on their underlying diseases. While it might seem that nausea and fatigue frequently occur together, these symptoms are reflections of an underlying condition, such as gastritis. Thus, they are expected to be conditionally independent when the disease is accounted for. Also, in questionnaire studies, the answers to questions are generally assumed to be conditionally independent given the underying mental cinditions.