Nonparametric Evaluation of Noisy ICA Solutions

The authors consider the noisy ICA problem. They first propose a numerical objective function that can be used as a guide to assess the quality of an existing ICA algorithm. This function is not suitable for optimization and therefore, the authors propose to use it in a meta-algorithm, where the purpose is to select the best ICA solutions out of several, where each solution is possibly produced by a separate algorithm. Then, they propose new contrast functions that are specifically suited for the noisy ICA problem. Finally, they study critical points of contrast functions so as to validate their use in noisy ICA problems. They provide numerical experiments to back up their claims.

This paper proposes a nonparametric score to adaptively pick the best noisy ICA algorithm from a set of candidates. This “independence score” is based on the characteristic function-based objective (CHF) introduced by Eriksson&Koivunen in 2003.

In practice, this independence score evaluates the inverse mixing matrix obtained by an ICA algorithm without requiring accessing to the true sources.

In addition, the paper proposes some new contrast functions and algorithms and present simulation results showing the effectiveness of the proposed independence score.

The presented paper focuses on the problem of noisy ICA which remains a significant challenge in classical machine learning. The authors introduces a nonparametric independent score extending the work in [21] to evaluate the estimation of the demixing matrices without requiring any prior knowledge of underlying noise distribution parameters. Further. the authors propose some new contrast functions and provides a very interesting discussion on the convergence for the presented contrast functions of noisy ICA which can be also applied for cumulant based contrast functions.

This work proposes a modification of the original CHFICA characteristic function to the noisy ICA case without requiring knowledge of the noise distribution parameters. The modified independence score is then used to select the best out of multiple ICA methods based on the score assigned to their solutions, in a per dataset basis. In practice, an average score is evaluated over random directions t, which renders the score computationally inefficient for direct Minimization. The empirical estimate of the new independence score is shown to converge to the population score if the data covariance is concentrated. Empirical evidence is shown that the new empirical score correlates well with the well-established Amari error.

This work also introduces two computationally efficient contrast functions to be Maximized (CHF-based and CGF-based) for BSS, both of which do not require higher moments and, thus, work well on data with near-zero kurtosis, like Bernoulli(p) with low p. Theoretical analyses of local and global convergence are also included for a large class of smooth contrast functions that meet certain desirable properties (Assumption 1). Properties (a) and (c) are critical as they yield contrast functions that are not influenced by additive independent Gaussian noise. The CHF contrast only requires a finite second moment and is suitable for heavy-tailed sources. The CGF contrast is not appropriate for heavy-tailed sources. Global convergence: The Hessian of both contrasts is shown to have the form C = BDB^T. The maxima of the Hessian-adjusted contrasts is shown to occur at B^-1 u = e_i for positive elements of D. Fast sequential power method optimization in a pseudo-Euclidean space defined by C^-1 was shown by [48]. Before optimization, C need only be computed once for some random vector u and just reused during the power iteration updates. A requirement of no sign change of the third derivative of the contrast over the half-lines is required to sufficiently distinguish between Gaussian and near-0 kurtosis source distributions. Local convergence: Linear convergence is attainable without the third derivative requirement. Geometric convergence is attained at some low epsilon based on how non-gaussian the source is. Contrary to prior work on cumulant-based methods, convergence of contrasts meeting Assumption 1 was established based on the characterization of the contrasts in the pseudo-Euclidean space and not on the convergence of the power method.

Experimental results support Meta's ability to select the optimal solution over various algorithms, showing variance reduction in the estimates, uniform performance across high and near-0 kurtosis. Evaluation of the CHF and CGF contrasts show that CHF outperforms other methods at higher noise, but only at sample size n > 20000. The Meta algorithm picks up the slack of CHF by selecting the competing PFICA solution when n < 20000.

We want to first thank all the reviewers for their valuable suggestions and insightful feedback. We believe we have addressed nearly all of their main technical questions. In what follows, we will address some important points each reviewer has raised. We will correct all the typographical issues pointed out and will not address them here.

Re: Clarity (Reviewer Dz1S)

Thank you for noting that our manuscript, proofs, and methodology are well-written and thorough. In the revised version of the manuscript, we will fix typographical and notational errors and add further details in the derivations to enhance readability and clarity.

Re: Gaussian noise assumption (Reviewer N8km)

The classical noisy ICA problem adds Gaussian noise to a mixture of non-gaussian independent components. This illustrious problem has applications ranging from signal processing and image analysis to biomedical data analysis. Nearly all the literature we found focuses on Gaussian noise. While it will be interesting to analyze the role of non-gaussian noise, that is outside the scope of this paper.

Re: Recovery of source signals (Reviewer c27m)

We address this concern in lines 60-65 of our paper. Under the noisy ICA model (Eq 1), both $\mathbf{x} = B(\mathbf{z}+\mathbf{g’}) + \mathbf{g}$ and $\mathbf{x} = B\mathbf{z} + (B\mathbf{g’} + \mathbf{g})$ for Gaussian $\mathbf{g}’$ are indistinguishable (see Voss et al (2015) pg 3). This makes recovery of the source signals impossible in the noisy ICA setting, even if the mixing matrix is known exactly because part of the noise could always be interpreted as part of the signal while preserving the model definition. Voss et al. (2015) propose learning the source signals by maximizing the Signal-to-Interference-plus-Noise ratio (SINR). Typically, in all related papers, algorithms are compared using the accuracy of estimating the $B$ matrix. But, with the estimated $B$ matrix one can obtain an SINR optimal estimation of the signals as in the previous citation, which we will clarify.

Re: Subgaussianity assumption in Theorem 2] (Reviewer c27m)

Yes, as long as the sample covariance matrix concentrates around the population covariance in operator norm, our proof holds. For example, [a] shows that if $\mathbb{E}[|X^T u|^q]\leq L^q$ is bounded for all unit vectors $u \in \mathbb{R}^{d}$ (Eq 1.6), then the covariance matrix concentrates at a rate $(\frac{d}{n})^{\frac{1}{2}-\frac{1}{q}}$. Under such distributions, which are more general than sub-gaussians, Theorem 2 will hold with a different error rate. We will clarify this. Our experiments include exponential source signals, which do not follow the subgaussian assumption. To answer your question, we have now added new experimental results (see pdf) with other super-gaussian sources (Laplace, Student’s t with 3, 5 degrees of freedom, and exponential). The Meta algorithm closely follows the best candidate algorithm even in the presence of many super-gaussian signals, which reflects that the sub-gaussianity assumption in Theorem 2 is not critical.

Re: Connection between the meta-algorithm and the new contrast functions] (Reviewer uqTT)

Most methods for noisy ICA center around designing an appropriate contrast function. Lines 87-103 in the paper, under “Individual shortcomings of different contrast functions and fitting strategies,” explain the different issues with different types of existing contrast functions. As explained in lines 104-110, this leads to two challenges: a) how do we adaptively pick an appropriate contrast function for the dataset at hand, and b) can we design better contrast functions? Our paper gives a full pipeline for a new adaptive algorithm for choosing from a set of contrast functions, which includes established ones and the new ones we introduce.

Re: Development of the meta-algorithm and the choice of randomization in its computation] (Reviewer uqTT)

Lines 115-124 build intuition for the score used in the Meta algorithm using the noiseless case. For the noisy case, one needs to account for the characteristic function of the Gaussian noise, whose covariance matrix is unknown. The second terms in JOINT and PRODUCT essentially cancel out the additional terms using estimable parameters, i.e. the covariance matrix $S$ of the data (estimated using the sample covariance).

Lines 148-151 discuss why we do not use the independence score as an optimization objective and instead use it to choose the best algorithm amongst candidates. This leads to the meta-algorithm in Algorithm 1.

Remark 2 explains the choice of randomization in the Meta algorithm. Essentially, all related papers that use the characteristic functions in ICA or for Gaussianity testing (see the citations in Remark 2, line 155) average over $t$ sampled from a spherical Gaussian.

Re: Origin and development of contrast functions (Reviewer Dz1S)

Section 3.2 (lines 163-168) provides the mathematical properties that are useful in a contrast function for designing provable optimization algorithms for ICA. Lines 169-172 explain what each of these properties imply. Section 3.3 then develops the new contrast functions that satisfy these important properties.

References:

[a] Vershynin, Roman. "How close is the sample covariance matrix to the actual covariance matrix?." Journal of Theoretical Probability 25, no. 3 (2012): 655-686.

This paper presents a modification of the noiseless characteristic function-based ICA to the noisy ICA without requiring the knowledge of the Gaussian noise distribution parameters. Two computationally efficient contrast functions are introduced in the paper. Reviewers claimed that the origin of these new contrast functions was not clear, but during the reviewer-author discussion period it has been clarified up to a certain point. One of serious concerns is that the paper seems to a collection of not clearly-developed ideas, rather than exploring coherently and clearly a single idea. I believe this is partially true. During the AC-reviewer discussion period, there was a mix of support and concern. One reviewer with negative comments agreed that he/she would not mind accepting the paper even though he/she wanted to keep his/her rating as a minority. Consider all the comments raised by reviewers, I think that paper has a contribution in the direction of noisy ICA. I would like to ask the authors to revise the paper carefully, improving the clarity in the final version.