A Generic Family of Graphical Models: Diversity, Efficiency, and Heterogeneity

We sincerely appreciate your time in reviewing our paper and your insightful comments. In the following response, we would like to address your major concern and provide additional clarification.

For weakness:

The author(s) write the family in a very general way, but the examples are more or less all Gaussian related. Therefore, there are some concerns about novolties, as the defined family may be some extensions of Gaussian and we can directly use results.

Thanks for your comment. First, since the focus of this work is primarily on graphical network inference, with the application scenario being gene regulatory network inference using biological genomics data, the examples provided here are mainly based on Gaussian distribution models in hierarchical settings. However, the new concept introduced—the marginally recoverable family—includes distributions beyond just Gaussian-related distributions. For instance, the multinomial distribution belongs to this family but does not Gaussian-related. Let $\mathbf{X}=\left(X_{1}, \ldots, X_{p}\right)^\top \sim Multinomial(n,\boldsymbol{\theta})$, where $\boldsymbol{\theta}=(\theta_1,\dots,\theta_p)^\top$. Then, for $1 \leq j < k \leq p$, we have $X_j \sim Multinomial(n, \theta_{j})$ and $\left(X_j , X_k \right)^\top \sim Multinomial(n, (\theta_j, \theta_k)^\top)$. This demonstrates that the multinomial distribution satisfies the definition of the family.

Secondly, our defined family is not an extension of the Gaussian distribution that can be addressed by existing methods. Even when the latent layer is modeled using a Gaussian distribution, distributions with hierarchical structures still pose significant challenges for parameter estimation. For example, as mentioned in the paper, there are few methods developed for estimating the precision matrix of the MPLN model in high-dimensional settings. Compared to Gaussian mixture models, the MPLN model presents a greater challenge during the parameter estimation process using the EM algorithm, as its expected complete log-likelihood involves high-dimensional integrals that are difficult to compute. Therefore, for such models, we cannot directly apply existing methods or results.

For other comments or suggestions:

Figure 1 is blank.

Thank you for your comment. We appreciate your attention to Figure 1. We would like to clarify that Figure 1 is not blank. It is possible that the white areas in the figure were confused with the background, which might have led to the impression that the figure is empty. To avoid this misunderstanding, we will add borders around each subplot in Figure 1 in the final version. We would like to clarify that in subplot (a), the white areas represent positions where the true network does not have edges. In subplots (b) to (e), the white areas at position $(i,j)$ indicate that the frequency of false positives (i.e., incorrectly predicting an edge where there is none) for that position is zero. We hope this explanation clarifies the meaning of the white areas.

We sincerely appreciate your time in reviewing our paper and your positive feedback. Here we address your comments as follows.

For weakness:

Marginal recoverability may not extend to distributions with higher moments.

Thanks for your comment. To facilitate an intuitive network representation, we impose certain restrictions on the parameters of the family, typically related to the first and second moments. However, motivated by the core ideas behind this family, we can provide a more general definition that extends to distributions determined by higher moments. Specifically, we introduce the definition of a $d$-marginally recoverable family. Let $\mathbf{X}$ be a $p$-dimensional random variable, and let $d$ be a constant such that $d<p$. We say that $\mathbf{X}$ is $d$-marginally recoverable if any $d$-dimensional marginal distribution of $\mathbf{X}$ belongs to the same distribution family, and the parameters of all $d$-dimensional marginal distributions collectively characterize the parameters of the full distribution. We will introduce this general definition in the Discussion section of our revision and explore additional distribution examples that align with it in our future work.

For suggestions:

1. Notation $d$ is used for both dimension and Hellinger distance in Condition 4.1 and 4.2.

Thank you for your helpful suggestions. We have corrected the duplicate symbols in the manuscript by replacing $d$ (used for dimension in Conditions 4.1 and 4.2) with $k$.

2. It is very hard to interpret the sample size requirement in Theorem 4.4. Some discussion of sample complexity depending on the dimension / sparsity will be appreciated.

For some $\eta >2$, the result in Theorem 4.4 holds under the sample size condition $n>O(\eta s^{1/2}\log p)$, where $n$ is the sample size, $s$ denotes the sparsity level of the network (i.e., the number of nonzero entries), and $p$ denotes the dimension. Since $\widehat{\Theta}_g$ recovers all the zeros and nonzeros in $\Theta_g$ with probability $1-p^{2-\eta}$, increasing $\eta$ raises this probability towards 1, but requires a larger sample size.

3. It would be helpful to provide an example with both continuous and discrete variables with explicit modeling formula.

Here, we provide a specific example involving both continuous and discrete variables, with an explicit modeling formula. Assume that $\mathbf{X}=(\mathbf{X}_1,\mathbf{X}_2)$, where $\mathbf{X}_1$ is a random $a$-vector and $\mathbf{X}_2$ is a random $b$-vector. Suppose that there is a random vector $\mathbf{Z}_1=(Z_1,...,Z_a)$ such that $(\mathbf{Z}_1,\mathbf{X}_2)\sim N(\mu,\Sigma)$ and $X_j=I(Z_j>C_j)$ for all $j=1, ..., a$, where $C_j$ is a constant. We can easily verify that $\mathbf{X}$ is marginally recoverable. While this paper does not primarily focus on mixed-type data, our approach can be applied to parameter estimation as long as the variables meet the marginal recoverability definition.

For questions:

1. How do we justify Condition 4.1 and 4.2 intuitively?

Conditions 4.1 and 4.2 establish a relationship between the parameter distance and the distance between the marginal density functions. Intuitively, for any two parameters within the parameter space, if their Euclidean distance is small, the corresponding Hellinger distance between their marginal density functions will also be small. When verifying that specific distributions satisfy Conditions 1 and 2, we found that this is equivalent to verifying the following conditions:

(A1) The Fisher information matrix is positive definite and continuous with respect to the parameters.

(A2) Both the score function and the Hessian matrix are dominated by integrable functions.

(A3) The parameter space is compact, and the model is identifiable.

Among these, (A1)-(A3) are standard regularity conditions. However, expressing the original conditions as (A1)-(A3) would appear somewhat redundant. Conditions 4.1 and 4.2, on the other hand, provide a more intuitive link between the distances of densities and parameters, making the formulation clearer and more accessible. Therefore, we use these boundedness conditions to present the core conditions of our theorem instead of (A1)-(A3).

2. Is there any theoretical guarantee for the consistency of membership assignment in the mixture modeling?

For the Gaussian mixture model, Chen and Zhang [1] derive minimax lower bounds for clustering with respect to the misclustering error rate. To the best of our knowledge, establishing theoretical guarantees for membership assignment remains a challenging problem. Due to time constraints, we have not addressed the consistency of membership assignment in general mixture models in this paper, but this could be explored in future work.

Reference

[1] Chen, X. and Zhang, A. Y. Achieving optimal clustering in gaussian mixture models with anisotropic covariance structures. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.

We sincerely appreciate your time in reviewing our paper and your insightful comments. In the following response, we would like to address your major concern and provide additional clarification.

For theoretical claims:

I believe that neither Theorem 4.2 nor Theorem 4.3 should be labeled as "Theorems". Instead, I would suggest calling them "Lemmas," as they directly follow from other established theorems.

Thank you for your valuable suggestion. Since these two theorems are derived from established results, we acknowledge that referring to them as "Lemmas" is more appropriate. Nevertheless, we would like to emphasize that these results establish that once a connection between the parameter distance and the density distance is made (as specified in Conditions 4.1 and 4.2), we can obtain tail probability control for high-dimensional parameters. We believe that this provides new insights and a novel proof strategy. Furthermore, verifying that a specific model satisfies the necessary conditions (Conditions 4.1 and 4.2) is nontrivial, especially for complex models such as MPLN. In fact, a substantial portion of the appendix (see Appendix A.3) is dedicated to rigorously demonstrating that the MPLN model satisfies these conditions.

For weaknesses:

While the family is presented in a general form, it appears to only include elliptical copula distributions (and simple compositions thereof). This raises some concerns about the novelty of the approach, as elliptical copula families are already well-established in the literature. Additionally, elliptical copulas are quite limited in their expressive power, particularly when it comes to modeling tail dependencies, which either vanish asymptotically or are entirely absent in the special case of the Gaussian copula. In many applications of copulas, this limitation has led to the development of more complex copula families. Therefore, the claim on page 2 that "The marginally recoverable parametric family includes many of the most common distributions" might be somewhat misleading.

Thanks for your comment. In fact, the family includes distributions beyond elliptical copula distributions (and simple compositions thereof). For instance, the multinomial distribution belongs to the family but does not fall within the class of elliptical copula distributions. Specifically, let $\mathbf{X}=\left(X_{1}, \ldots, X_{p}\right)^\top \sim Multinomial(n,\boldsymbol{\theta})$, where $\boldsymbol{\theta}=(\theta_1,\dots,\theta_p)^\top$. Then, for $1 \leq j < k \leq p$, we have $X_j \sim Multinomial(n, \theta_{j})$ and $\left(X_j, X_k\right)^\top \sim Multinomial(n, (\theta_j,\theta_k)^\top )$. This demonstrates that the multinomial distribution satisfies the definition of the family.

Furthermore, the concept of the marginally recoverable family can be extended to a more general framework. We introduce the definition of a $d$-marginally recoverable family as follows:

Let $\mathbf{X}$ be a $p$-dimensional random variable, and let $d$ be a constant such that $d<p$. We say that $\mathbf{X}$ is $d$-marginally recoverable if any $d$-dimensional marginal distribution of $\mathbf{X}$ belongs to the same distribution family, and the parameters of all $d$-dimensional marginal distributions collectively characterize the parameters of the full distribution.

In this way, the marginally recoverable family becomes more flexible. Under this definition, the multinomial distribution mentioned earlier is actually 1-marginally recoverable, as only the parameters of the 1-dimensional marginal distributions are required to recover all the parameters of the full distribution. We will introduce this general definition in the Discussion section of a future revision and explore additional examples that align with this broader definition in our future work.

For modeling tail dependencies, we acknowledge that elliptical copulas do have limitations in this regard. That said, this concern is primarily relevant in financial applications. Our work, on the other hand, focuses on network inference and its applications to biological genomics data, where the elliptical copula family is commonly used, and tail dependencies are generally not a central issue. Exploring tail dependencies within our framework is an interesting direction for future research. The claim on page 2 could be revised to: "The marginally recoverable parametric family includes distributions commonly used in the field of bioinformatics," which would be a more precise statement. We sincerely appreciate your constructive suggestion.

We sincerely appreciate your time in reviewing our paper and your insightful comments. In the following response, we would like to address your major concern and provide additional clarification.

For your major concern in the claims and evidence section, as well as the similar concern raised in the theoretical claims and weaknesses sections:

The theoretical guarantees hinge on two stringent conditions (the lower-boundedness and upper-boundedness conditions in Conditions 4.1 and 4.2). Their restrictive nature may limit applicability in practical scenarios where the data do not closely conform to these idealized requirements.

Thank you for your comment. We appreciate your concern about the potential restrictiveness of Conditions 4.1 and 4.2 and their applicability to real-world data. We would like to clarify that Conditions 4.1 and 4.2 are, in fact, relatively mild. As demonstrated in Appendix A.3.4 and Appendix A.4, both the MPLN distribution and the latent Gaussian copula model for binary data satisfy these conditions.

More specifically, our proof establishes that a distribution satisfies Conditions 4.1 and 4.2 as long as the following assumptions hold:

(A1) The Fisher information matrix is positive definite and continuous with respect to the parameters.

(A2) Both the score function and the Hessian matrix are dominated by integrable functions.

(A3) The parameter space is compact, and the model is identifiable.

Among these, (A1)-(A3) are standard regularity conditions, which are widely used to ensure the asymptotic efficiency of the maximum likelihood estimator, as shown in Shao's book [1].

For the other weakness:

More experiments showing robustness under deviations from the assumed conditions would be valuable.

Thank you for your comment. Due to rebuttal time constraints and the fact that we have not identified any marginally recoverable distributions that do not satisfy the conditions, we have not conducted additional experiments to demonstrate robustness under deviations from the assumed conditions at this time.

Reference

[1] Shao, J. Mathematical statistics. Springer Science & Business Media, 2008.