DipDNN: Decomposed Invertible Pathway Deep Neural Networks

We would like to thank you for your time and effort in reviewing our paper. We are pleased you recognize the proposed method's applicability to diverse inverse problems. The suggestions you provided for clarifying the paper's target and contribution are very useful, and we will use them for revision. Below, please find our responses to the questions.

Specify objectives of the inverse problem:

Define the inverse problem:

For system identification, the target is to learn a mapping $f: \mathcal{X} \rightarrow \mathcal{Y}$ from data for estimating $y$ given $x$. The inverse problem is an opposite direction of system identification to infer states $x$ given $y$. Therefore, our target is first to learn a forward mapping and then derive the inverse counterpart. Mathematically, the objective function includes 1) a regular supervised learning error term of the forward learning and 2) a reconstruction error term of using the inverse mapping to recover back to $x$.

Clarify the mathematical objectives:

Thanks for pointing out the typo in the math expression. We checked it and found that the previous objective and description may cause confusion. We rewrite it here.

As computing the inverse solution is the final target, we define the objective using the inverse mapping $g: \mathcal{Y} \rightarrow \mathcal{X}$.

${g}^* = \operatorname*{argmin}_{g \in \mathcal{G} } \sum_{i=1}^N \ell_1\left({g_{\theta}}^{-1}\left(\bm{x}_i\right),\bm{y}_i\right) + \sum_{i=1}^N \ell_2\left(\bm{x}_i, g_{\theta}\left({g_{\theta}}^{-1}\left(\bm{x}_i\right)\right)\right)$

The first term is the supervised learning loss to minimize the mismatch in forward model recovery, where $\ell_1(\cdot)$ denotes the mean square error we used in this paper. The second term is the reconstruction error to make the inverse consistent with the forward. $\theta$ are neural network parameters of the forward model, which is optimized during training. In our proposed method, we design analytical invertibility in the forward NN model such that the inverse counterpart shares the parameters $\theta$.

Therefore, once the forward model is well-trained, we directly use the invertible NN to derive the inverse model and use it for inference of $x$. No more training is needed. This is also why an analytical inverse form is essential; otherwise, the inverse computation is challenging.

Clarify components in Figures 1 and 2:

We use Figure 1 to illustrate the architecture limitation and representation power of existing invertible networks. They motivate us to design the proposed DipDNN in Figure 2. Figure 1(a) is a basic layer of the classic NICE model [1]. The blue component denotes the nonlinear NN correlation, which is only between $x_{I_1}$ and $y_{I_2}$. ${x}_{I_1}$ and ${y}_{I_1}$, ${x}_{I_2}$ and ${y}_{I_2}$ are linearly correlated and $x_{I_2}$ and $y_{I_1}$ are not correlated. Therefore, at least three such layers are required to enable full nonlinear coupling of all inputs with all outputs in Figure 1(b). Regular NN takes one layer for full coupling, so we try to compress Figure 1(b) with an equivalent representation and compare the network connection with a regular NN. Red dots denote Leaky ReLU activation, providing nonlinearity on the linear paths in the reduced architecture. Leaky ReLU activation is a strictly monotone function customized from ReLU, which preserves invertibility in nonlinear mapping.

Figure 2 presents the proposed decomposed-invertible-pathway DNN (DipDNN) model. The blue lines indicate connections in NN, and the red dots still denote Leaky ReLU. Here, we first maintain the dense NN architecture and then enforce one-to-one correspondence by decomposing the nested layer of NICE into a basic invertible unit, as shown in the top right corner of Figure 2. Subsequently, we extend the invertible unit to wider and deeper NN.

First, we would like to thank the reviewer for the positive review. We’re pleased you found our work meaningful and well-designed and our theoretical analysis solid. The suggestions are very valuable, and we’ll use them for revision. Below, we provide responses to the questions.

Regarding baseline methods

The reviewer mentioned a very important aspect of choosing baselines to compare with. The related work is quite limited because the target problem is inverse point estimates, especially in providing analytical inverse. Most work focuses on a generative setting for inverse problems in image-related tasks, e.g., flow models. We select [1] for the classic bijective function using affine coupling. Its latest variation is [2] with applications on discriminative learning tasks, and we refer to their training details in experiments. I-Resnet [3], mentioned by the reviewer, is another typical design of invertible NN with Lipschitz regularization. To the best of our knowledge, the following work has not changed the core invertibility designs of [1] and [2] in NN architectures, which is our focus. So, we choose them as the baselines. We would really appreciate it if the reviewer had any recommendations on an up-to-date baseline method.

Considering complex dataset and model scalability

We add some experiments to test the model on CIFAR-10 and compare it with i-Resnet. We test a CIFAR-10 classification task and note that the forward-inverse mapping is built between inputs to the last feature extraction layer, where a linear classification layer is used on top of the invertible blocks. Due to the time limit, we reduce the hidden layers and control all the models to have the same number of layers (modes using MLP and having the dimension be 3072) for a fair comparison.

Clarify NN architectures:

First, we modify the baseline models for each testing case to control all the NN architectures to be as similar as possible for fair comparison. For example, NICE has a nested DNN in the affine coupling layer, and we use the same MLP for the nested DNN as our DipDNN. Then, the numbers of hidden layers differ from case to case, e.g., 3 layers for face completion task and 6-8 layers for MNIST based on cross-validation. For state estimation, it depends on the problem size, e.g., 3 layers for IEEE 8-bus network and 6-10 layers for IEEE 123-bus network based on cross-validation.

Insignts on the scalability:

For the scalability of the model, we agree with the reviewer that more practical large-scale use cases will be helpful. In existing work, the invertible model is often used for density estimation, for which image-related tasks are usually large-scale. Our work focuses on general inverse problems in physical systems, for which the scale varies from case to case. Unlike generative learning in density estimation, the inverse problem in physics targets the point estimates and usually has a critical requirement for accuracy. Thus, the emphasis of this paper is to obtain an accurate inverse solution. The current model has been tested on small synthetic examples (2-9 variables) cases as large as a 123-bus power system for physical system state estimation and MNIST data set (28$\times$28 = 784) for image construction. During our experiments, we have some observations regarding scalability. In inverse problems, a typical scalability challenge is that, with a larger problem size and larger model, the numerical errors will aggregate and lead to numerical non-invertibility. We show partial results in Figure 4 for the error propagation with respect to increasing problem dimension and nonlinearity and increasing model depth. The errors increase on an exponential scale. For this challenge, we analyze with details in Appendix A.4. While large-scale problems may aggregate errors, we can regularize bi-Lipschitz continuity to enhance inverse stability.

Thank you for taking the time to review and provide valuable comments on our work. We’re pleased to see that you found our work novel, and we appreciate your suggestions on the presentation of the paper's contents and contributions. We’ll include them in our revision. Below, please find our response to the questions.

Clarify the contributions of our work.

To clarify the paper's contributions, we will divide the explanation into the drawbacks of existing methods and designs to address them.

General motivations from existing work:

The neural network is not invertible in nature because the multi-layers of interconnections break a strict one-to-one correspondence. One group of previous methods approximates the numerical inverse, i.e., minimizing reconstruction error. It is difficult to ensure an accurate inverse, especially for extrapolation. For analytical inverse, several methods try to embed invertibility into the NNs, and we analyze two typical methods. One is i-ResNet [1], which has proved invertibility with regularized weights, but there is no analytical inverse form. Still, we’ll refer to its regularization to address numerical issues. The other is the NICE model [2], where the affine coupling in layers (and also the exponential nonlinearity designed later in [3]) is strictly bijective and has an easy-to-derive inverse form. We started with applying this model to the inverse problem in physical systems and found poor performance in that 1) the maximize likelihood loss function doesn’t apply as our task is discriminative learning and the data is not generated from a Gaussian distribution, 2) the random splitting of image pixels is not fair for variables with physical meaning as different splitting groups lead to different results, 3) each layer ensures the invertibility via linear paths and bypasses nonlinear mappings.

Illustrate the drawbacks of nested design and fixed input/output splitting:

Points 2) and 3) are related to the architecture limitation and representation power of existing invertible networks. To illustrate with an intuitive example, we would like to point the reviewer to Appendix A.1, where Page 14 includes three figures to compare the network architectures. Figure 9(a) is a basic layer of NICE. The nonlinear NN correlation is only between $x_{I_1}$ and $y_{I_2}$. $x_{I_1}$ and $y_{I_1}$, $x_{I_2}$ and $y_{I_2}$ are linearly correlated and $x_{I_2}$ and $y_{I_1}$ are not correlated. Therefore, at least three such layers are required to enable full coupling of all inputs with all outputs in Figure 9(b) (this can be inferred from Jacobian derivation on Page 13). Regular NN takes one layer for full coupling, so we compress Figure 9(b) into Figure 9(d) with an equivalent representation and compare the network connection with a regular NN. In Figure 11, we can observe that, while all the dimensions are coupled, the nonlinear mapping is separated into groups and isolated within each group coulping, i.e., from $x_{I_1}$ to $y_{I_1}$, from $x_{I_2}$ to $y_{I_2}$, from $x_{I_1}$ to $y_{I_2}$, and from $x_{I_2}$ to $y_{I_1}$. Many interconnections are eliminated, like nonlinear couplings of variables in $x_{I_1}$ and variables in $x_{I_2}$.

Therefore, we see that the nested design in the NICE model ensures invertibility at the cost of $\sim 1.5\times$ computation complexity (i.e., at least three layers for full coupling) and representation power (i.e., an equivalent sparse representation of regular NN). This is why we are motivated to design an invertible model that relaxes the fixed splitting of variables and maintains regular NN’s representation power as much as possible.

Proposed designs to address the drawbacks:

In the proposed decomposed-invertible-pathway DNN (DipDNN) model, we first maintain the dense NN architecture and then enforce one-to-one correspondence by decomposing the nested layer of NICE into a basic invertible unit, as shown in the top right corner of Figure 2. The adoption of Leaky ReLU activation maintains invertibility in nonlinear correlation. In this way, we could prove the preserved approximation capability using the LU decomposition of weights and the latest theory of narrow deep networks [4].