Continual Nonlinear ICA-Based Representation Learning

We are very grateful for your time, insightful comments, and encouragement. Below please see our point-by-point response.

It is not clear how you address continual learning.

Thank you for your question. As detailed in Section 3.3, we address continual learning through pursuing two objectives: (1) the reconstruction of observations within the current domain and (2) the preservation of reconstruction capabilities for preceding domains. Specifically, we use GEM algorithm to enforce that the movement of network parameters learning a new domain should not result in an increased loss for the previous domains.

What is the actual form for the domain variable $u$?

$u$ is an indicator of domain which can be represented with a vector or a scalar.

This paper only considers a general continual learning setting, which relies on the task information. However, the proposed approach can not be used in task-free continual learning.

Thank you for your question and we would like to re-emphasize the difference between ours with traditional continual classification tasks. In the traditional continual classification task, each task is independent. The purpose of continual learning is to make the model remember each task as much as possible. In this situation, not knowing the identities of tasks will indeed affect testing. However, for our identifiable nonlinear ICA learned in sequentially arrived domains, we need multiple domains to make the model identifiable. Once we have enough domains, knowing the identity of the domain (task) or not is not important at all for the test phase. Regardless of which domain the tested data comes from, even a completely unseen domain, we can still recover meaningful latent variables (component-wise identifiable). Figure 5 (a) also validates this point as the model is always tested on all 15 domains while the training data increases gradually from 1 domain to 15 domains.

For the training phase, it should be noted that our observations are generated from changing distribution through fixed transformations. Thus, it would be easy to capture the boundary of each domain for model training.

The theoretical framework is based on the existing work [1].

Thanks for your comment. Compared with the framework from [1], we introduced the subspace identifiability for changing variables, which is essential to investigate the case where few domains are observed. Beyond that, we have demonstrated that, in the context of continual learning, the identifiability of certain variables becomes compromised with the introduction of new domains, as outlined in Proposition 2 and empirically shown in Figure 6.

The number of baselines in the experiment is small.

The reason we have not conducted comparisons of our method with other nonlinear ICA approaches is due to the distinctive structure of our model. Specifically, our approach involves partitioning latent variables into two distinct categories: changing and invariant components. Thus, other nonlinear ICA frameworks cannot be directly compared as a baseline.

More continual learning experiments should be performed.

We are currently conducting the experiment on image datasets and will report the results once we get them.

The lifelong generative modeling experiments should be provided.

We appreciate your comment, but we would like to clarify our understanding of your question to provide a more accurate response. Section 4 of our paper is dedicated entirely to the lifelong generative modeling experiment. If you could provide more specific details or context regarding your query, it would greatly assist us in addressing your concerns or questions more effectively.

[1]. Kong. et al. Partial Disentanglement for Domain Adaptation, 2022

Dear authors, thanks a lot for your comments. I will go through your comments regarding the theory in more detail (from skimming them, they seem to address many of my open Qs).

I wanted to quickly chime in and confirm that additional (convincing) empirical results would be the main reason for me to change my evaluation of the paper. I appreciate that you decided to run additional experiments, and am happy to engage in further discussion once first results are in.

We sincerely thank the reviewer for the time dedicated to reviewing our paper, the constructive suggestions, and encouraging feedback. Please find the response to your comments and questions below.

In the derivation in A1 and the following sections, the arguments to the Jacobians, e.g. $J_h$, are dropped. Is $u$ related to $J_h$?

Thank you for your question. $u$ is not related to $J_h$. The transformation $h = g^{-1} \circ g$ describes the relationship between the true latents $z$ and the estimated variables $\hat{z}$, which will not be influenced by the change of domains $u$. For different $u$, Jocabian $J_h$ is the same and the existence of $u$ is to help us to remove the intractable term $J_h$. We will make this clear in the revised manuscript.

Let us summarize the overall logic of the proof: we can easily construct the relationship between estimated variables $z$ and true latents $\hat{z}$ like Eq(16) as both $g$ and $\hat{g}$ is invertible. The challenge is that we do not know about $J_h$ (once we know, everything is solved), while we want to investigate the relationship between $\hat{z}$ with $z_s$. To solve the untractable term $J_h$, we introduce the domain $u$ and produce multiple equations like Eq~(17). Then, we can remove the intractable Jocabian term by taking the difference for every equation with the equation where $u=u_0$. This is why the $u$ is not related to $J_h$.

"the distribution estimate $\hat{z}_j$ variable doesn't change across all domains" and "Similarly, q_i remains the same for"

The reason that why $\hat{z}_j$ doesn't change across all domains is as follows: the $\hat{z}_j$ is defined to be the estimated invariant variable that will not be affected by domain $u$, for $j \in [n_s+1, \dots, n]$. Similarly, $q_i(z_i, u)$ for $i \in [n_s+1, \dots, n]$ represents the log density of groundtruth invariant variables, and thus it also does not change across all domains.

Is there a reason why $u$ is assumed to be a vector? for the purpose of the proof, isn't it sufficient to assume an integer value?

Thanks for your insightful observation. We totally agree with you that it is sufficient to assume an integer value for $u$ for the purpose of the proof. In our framework, $u$ serves as an indicator to denote different domains, and it can indeed be represented either as a vector or a scalar. We opted for a vector representation to maintain generality (when dimension reduces to one, a vector degrades to a scalar). We will update the manuscript to include this discussion.

between Eqs (21), (22), is it necessary to keep the argument $0$ ?

Thanks for your suggestion which helps improve the presentation and lighten the notations. We have updated the manuscript to remove the argument $0$.

The error bars are quite large between baseline and joint --- did you run a test whether the improvements observed are significant?

We appreciate your insightful observation. All experiments shown in Figure~5 are run with seeds set at 1,2,3. The error bars of baselines are indeed large--a possible reason is that the performance of the baseline is heavily influenced by the last domain, giving rise to the large variance. In contrast, the error bars or variances of our proposed method and the joint approach are much smaller, which further indicates the importance of a continual learning approach in this setting. We will include this observation and discussion in the experiment section.

Performance of the empirical validation is far from optimal, MCCs are substantially smaller than 1. What would it take to observe a result close to the "theoretical limit"?

Thanks for your question. A possible reason is that the estimation procedure involves neural networks, and the overall optimization problem is nonconvex, as is typical for methods involving deep learning. Thus, the estimation process, even for the joint training approach, may never achieve the global optimum. Another reason is that we divide the latent variables into invariant and changing parts, which may be more challenging than regular nonlinear ICA where all latents are changing. In fact, similar observations have been reported in [1].

Have you considered how different components (number of latents, number of samples in the dataset, ...) influence the final result?

Thanks for your question. Kindly refer to Figure 4 in the paper which shows the influence of different components ($n_s=4,n=8$ and $n_s=2, n=4$) on the final MCC results. We also conducted an ablation study to investigate the influence of $\hat{n}_s$ as shown in Table~1.

For different number of samples, in light of your suggestion, we conducted additional experiment results. For the case of $n_s=4,n=8$, we conducted training using 5000 samples and testing with 1000 samples for each domain. For the scenario of $n_s=2, n=4$, we conducted training using 10000 samples and testing with 1000 samples for each domain. We will add these details in the appendix of the final version.

We sincerely thank the reviewer for the time dedicated to reviewing our paper, the constructive suggestions, and encouraging feedback. Please find the response to your comments and questions below.

You claim that "[Nonlinear ICA] still relies on observing sufficient domains simultaneously...". -- Not true necessarily: for example [1], shows the identifiability of hidden Markov nonlinear ICA. As is well known, HMMs can be learned sequentially / in online fashion with the latent states analogous to different domains.

Thank you for your insightful observation. It is indeed correct that Hidden Markov Models (HMMs) can be learned sequentially. However, as delineated in reference [1], the identifiability of the model is contingent upon the concurrent observation of a sufficient number of latent states or domains. On one hand, the validity of Equation (22) is compromised in scenarios where the model experiences 'forgetting', particularly when only a single latent state is observed at any given time. This scenario presents a stark contrast to the framework and contributions of our study, which focuses on a genuinely continual learning approach. On the other hand, to preserve the identifiability of the model in a sequential learning context (as proposed in [1]), where only one latent state is observed at a time, the algorithm we have used in our paper could be adapted as it only focuses the optimization procedure.

it seems to already to cover mostly what is considered in the identifiability theorems here, and provides stronger results in a more general setting. It is therefore important to contrast to this work and explain what is novel here.

Thank you for your feedback. We agree that [1] and [2] are highly relevant, which we will discuss further in the revised manuscript. At the same time, we would also like to point out the differences with our setting (and therefore one is not "stronger" than another):

(1) While [2] is predicated on exploiting the temporal structure of latent variables, our paper delves into a scenario where the latent variables are independently and identically distributed (i.i.d) within each domain. This distinction is crucial as the reliance on the use of temporal structures in [1] and [2] can indeed facilitate stronger identifiability results.

(2) Our papers separate the latent variables into invariant and changing part, which is practical in real-world scenarios. We thoroughly investigated the conditions under which these two groups can be segregated (requiring at least $n_s+1$ domains) and also provided the sufficient condition that each changing variable is identified up to a nonlinear transformation ($2n_s+1$ domains). This exploration into the separability and identification of latent variables underscores the novelty and depth of our study.

(3) In contrast to [1], which presupposes the conditional distribution of latent variables within the exponential family framework, our work does not make such an assumption.

Other important and similar work is also ignored and need to be discussed such as [3]-[6]. Note also that most of these works do not assume auxiliary variables.

Thank you for sharing the references. We have provided a brief discussion on related works that do not assume auxiliary variables; see the third paragraph of Section 1. Given your suggestion, we will cite and discuss [3]-[6] in the revised version of our paper. We will also modify the sentence "it still relies on observing sufficient domains simultaneously" to "many of them still rely on observing sufficient domains simultaneously", to avoid any possible confusion.

The paper frames itself as "Causal Representation Learning" but it seems like it's much more related to nonlinear ICA and doesn't learn latent causal relationships -- see [6] and [7]. I recommend the authors to reconsider the use of this term.

Thanks for your suggestion. We have reconsidered the use of the term and will use "continual identifiable representation learning" instead.