MetaInv: Overcoming Iterative and Direct Method Limitations for Inverse Learning

We appreciate the reviewer’s concerns regarding the motivation of our work and the selection of specific models for analysis. Based on previous responses, we clarify how this study narrows its scope, justifies the choice of models, and provides theoretical and empirical contributions to inverse problems.

This study focuses on a specific class of inverse problems requiring deterministic two-way mappings for tasks such as physical system state estimation, image classification and recovery, and adaptive control. Unlike probabilistic setups or generative learning methods, which aim to model multimodal distributions or perform density estimation, we aim to recover point estimates with forward-inverse consistency. This focus prioritizes precision and interpretability, making explicit invertibility a central requirement for the model design.

We agree with the reviewer that there is a general difference between image and physics data. However, other than the data type, more factors and scenarios need to be considered when solving inverse problems:

• Data types and inverse learning tasks: Even using the same image data (e.g., MNIST and CIFAR10), there can be significantly different tasks. A major group of existing works on inverse problems focuses on density estimation tasks, which aim to generate complex distributions of images and fall under generative learning in a probabilistic setup. However, there are also image classification and recovery tasks, which belong to discriminative learning and require precise point estimates. Similarly, physical systems involve both stochastic modeling to quantify uncertainties and deterministic inverse/two-way mappings to estimate precise system states.

• Application-specific factors: Beyond data type, specific characteristics of application cases play a critical role in model performance. These include but are not limited to: i) the complexity of the underlying mapping, ii) the information redundancy of inputs, and iii) the precision requirements for forward and inverse predictions. Theorems 1 and 2 analyze the limitations of i-ResNet for i). Theorem 3 analyzes the transformation equivalency for ii), as the convolution layer compresses information. These are validated by Fig. 3, Fig. 9, Fig. 12, and Table II using different kernels of convolutions (different information compressions). The three metrics in Sec. 3.1 are defined to quantify for iii). By analyzing these diverse aspects comprehensively, our study evaluates the trade-offs of i-ResNet and DipDNN both theoretically and empirically.

We appreciate the reviewer for pointing out the writing issues, and provide responses to the questions below.

Question on MetaInv: The algorithm was included in the Appendix due to the space limit. We have moved it back to the main part and reduced the less important contents.

Question on Sec. 3.1: We have not only made the corrections but also rewritten Sec. 3.1 for clarification and better readability. The revision includes:

Inverse problems span numerous applications that involve recovering original variables from observed outputs. This work focuses on inverse learning through invertible mapping recovery, which is for point estimates of images or physical system states. The problem is typically formulated as approximating a forward mapping $f_{\theta}: \mathbb{R}^n \to \mathbb{R}^n$, where $y = f_{\theta}(x)$ is invertible. The goal is to find the relative inverse mapping $g_{\vartheta}$ such that $x = g_{\vartheta}(y) = f_{\theta}^{-1}(y)$, ensuring consistency with the forward process. The demands for two-way mapping rule recovery are distinct and varied even in one task. Thus, this work evaluates them using the following performance metrics:

• Forward Prediction Error (Fwd): Same as common one-way learning, it measures the ability of the model to predict $y$ from $x$. The invertible model is trained by minimizing the forward prediction loss (any discriminative learning loss $\ell_{\text{fwd}}$): $f^* = \arg \min_{\theta}\ell_{\text{fwd}}$ $({y},f(x))$

• Inverse Reconstruction Error (Inv): The reconstruction error evaluates the model's invertibility to recover the original inputs (where $\ell_{\text{inv}}$ is the mean square error for point estimates): $\ell_{\text{inv}} (x, g_{\vartheta} (f (x)))$.

• Inverse Consistency (Inv-Consist): It assesses the consistency between forward and inverse mappings by comparing the inverse predictions with the true labels ${y}$, rather than just the forward outputs: $\ell_{\text{fwd}} (x, g_{\theta} (y))$.

It is important to note the distinction between inverse accuracy and consistency. Inverse accuracy, or reconstruction error, indicates only the model's invertibility to compute the inverse, where analytical invertibility yields an error-free result, and numerical invertibility minimizes round-off errors. Consistency, on the other hand, evaluates the model's capability of two-way learning. It reveals the precision of global inversion. For example, minimizing reconstruction error doesn't necessarily need a low forward approximation error, while the consistency error integrates errors from both forward and inverse processes. Most works involving approximate one-to-one mappings focus on inverse accuracy alone, such as in image classification and recovery.

Question on Fig. 8:

We provided a detailed explanation of Fig. 8 (now Fig. 7 in the revised manuscript) below and modified the caption in the paper correspondingly.

Fig. 7 presents the performance comparison of different methods (ResNet, i-ResNet, NICE, DipDNN, and MetaInv) on the power flow case. The dataset is derived from an 8-bus power system, where load inputs ($x$) are used to estimate voltages ($y$) at different buses for forward mapping, and the inverse is the recovery of load conditions given voltages. Accurate inverse learning is essential here for recovering underlying system physics and ensuring consistency in power flow analysis.

The top-left plot shows forward voltage predictions (Fwd), and the two bottom plots show the inverse load recovery for both reconstruction and consistency (Inv and Inv-Consist). The table (top-right) provides quantitative error metrics (Fwd, Inv, and Inv-Consist) for the evaluated methods. While i-ResNet shows tightly bounded errors due to its Lipschitz constraint ($\text{Lip}<1$), its voltage predictions (top-left) reveal a pattern that diverges from the ground truth. This is attributed to the contractive property, which limits pointwise accuracy.

In contrast, DipDNN achieves better fits for both forward and inverse tasks. DipDNN, selected by MetaInv in this case, captures the system's essential behaviors more effectively, as evident from the closer alignment of its predictions with the ground truth. The inverse consistency plot (bottom-right) highlights DipDNN and MetaInv’s superior performance in achieving forward-inverse alignment compared to i-ResNet. However, excessive contraction, such as from convolutional layers, can distort inverse mappings, especially in systems with low redundancy.

Comment: Why this work remains significant?

We appreciate the reviewer’s concerns regarding the weakness of existing methods addressed in our study. Based on the previous response, we clarify how this work narrows the scope and provides theoretical analysis for a unique contribution to inverse problems.

This study focuses on a specific class of inverse problems requiring deterministic two-way mappings for tasks like physical system state estimation, image classification and recovery, and adaptive control. Unlike probabilistic setups or generative learning (e.g., MDNs, cINNs), where the objective often involves multimodal distribution density estimation, the goal here is to recover point estimates with forward-inverse consistency. Such problems prioritize precision and interpretability, necessitating explicit invertibility in the model design.

Thus, existing works that exhibit invertible model architectures are central to our work, where two key architectures are:

• Affine coupling layers (used in NICE and its extended versions): These layers provide analytical invertibility and computational efficiency, but the approximation capability is constrained by their reliance on affine transformations and split variables. DipDNN relaxes the limitations while maintaining the analytical inverse.

• Residual layers with Lipschitz constraint (used in i-ResNet): These are built upon the residual network for forward approximation capacity. Unlike analytical inverses, they ensure invertibility by enforcing Lipschitz continuity. The inverse requires iterative computation using a fixed-point algorithm.

By focusing on DipDNN and i-ResNet, we provide a rigorous analysis of foundational invertible architectures, rather than providing exhaustive experimental comparisons across all existing methods.

For both theoretical analysis and empirical implementation, we identify the trade-off of the two invertible models for various deterministic inverse problems. This is why MetaInv is introduced to supplement the capabilities of the two representative approaches for arbitrary tasks. By building on and improving DipDNN and i-ResNet, MetaInv provides a unified approach that achieves consistent forward-inverse performance while maintaining computational efficiency.

Comment: Why several inverse learning methods were omitted?

The literature on inverse modeling spans a broad spectrum, covering diverse applications and inverse models, as the reviewer highlights with the additional references [1]-[4]. Our study focuses on deterministic two-way/inverse mapping for inverse problems. Specifically, we target the invertibility design of models for tasks requiring point estimates, such as physical system state estimation, image classification and recovery, and adaptive control. These inverse problems differ from generative tasks like image density estimation, which form many previous works in the computer science community. Consequently, we evaluate invertible models' performances in discriminative learning, emphasizing forward approximation, inverse reconstruction, and forward-inverse consistency.

The methods mentioned by the reviewer either rely on stochastic modeling for image generation or lack explicit invertibility. Below, we discuss these methods in detail and justify their limited relevance to our work:

Genetic Algorithms
Genetic algorithms are widely used in optimization and design problems but are computationally intensive and often impractical for high-dimensional nonlinear inverse tasks. Additionally, they do not inherently provide invertibility or forward-inverse consistency, making them unsuitable for the deterministic two-way mappings we aim to achieve.

Conditional Invertible Neural Networks (cINNs) [1]
Conditional Invertible Neural Networks (cINNs) extend NICE and RealNVP models by enforcing invertibility through affine coupling layers. These models are primarily designed for generative tasks, using maximum likelihood principles to improve image density estimation. While cINNs perform well in generative tasks, they are less suited for tasks involving point estimates. Furthermore, we have reviewed, analyzed, and tested the NICE model. Since cINNs' analytical invertibility relies on affine coupling layers, similar to NICE, the comparison of the targeted inverse problem has been covered in this work.

Mixture Density Networks (MDNs) [2,3]
Mixture Density Networks (MDNs) are designed for probabilistic inverse modeling, where the outputs are characterized by multimodal distributions. These models preserve tractable probability density and involve uncertainty quantification, but they are not applicable to recover deterministic mappings that ensure one-to-one consistency between input and output. Furthermore, MDN models do not inherently enforce invertibility, which is central to our approach.

Tandem Models [4]
The Tandem models are specifically designed to solve inverse design problems, e.g., for optical materials. They focus on two-stage optimization, coupling an inverse-design network with a pre-trained forward model, using the latter to supervise the training of the former. Usually, the forward model is known, and the two models have different architectures. Although the general scheme involves inverse modeling, no invertibility is enforced in the Tandem structure. However, since Tandem models also target point estimates, such an inverse design problem could be an application case of our targeted inverse problem.

Comment: Mathematical typos from lines 139-155.

Thanks for pointing out mathematical typos. We have not only made the corrections but also rewritten Sec. 3.1 for clarification and better readability. The revision includes:

Inverse problems span numerous applications that involve recovering original variables from observed outputs. This work focuses on inverse learning through invertible mapping recovery, which is for point estimates of images or physical system states. The problem is typically formulated as approximating a forward mapping $f_{\theta}: \mathbb{R}^n \to \mathbb{R}^n$, where $y = f_{\theta}(x)$ is invertible. The goal is to find the relative inverse mapping $g_{\vartheta}$ such that $x = g_{\vartheta}(y) = f_{\theta}^{-1}(y)$, ensuring consistency with the forward process. The demands for two-way mapping rule recovery are distinct and varied even in one task. Thus, this work evaluates them using the following performance metrics:

• Forward Prediction Error (Fwd): Same as common one-way learning, it measures the ability of the model to predict $y$ from $x$. The invertible model is trained by minimizing the forward prediction loss (any discriminative learning loss $\ell_{\text{fwd}}$): $f^* = \arg \min_{\theta}\ell_{\text{fwd}}$ $({y},f(x))$

• Inverse Reconstruction Error (Inv): The reconstruction error evaluates the model's invertibility to recover the original inputs (where $\ell_{\text{inv}}$ is the mean square error for point estimates): $\ell_{\text{inv}} (x, g_{\vartheta} (f (x)))$.

• Inverse Consistency (Inv-Consist): It assesses the consistency between forward and inverse mappings by comparing the inverse predictions with the true labels ${y}$, rather than just the forward outputs: $\ell_{\text{fwd}} (x, g_{\theta} (y))$.

It is important to note the distinction between inverse accuracy and consistency. Inverse accuracy, or reconstruction error, indicates only the model's invertibility to compute the inverse, where analytical invertibility yields an error-free result, and numerical invertibility minimizes round-off errors. Consistency, on the other hand, evaluates the model's capability of two-way learning. It reveals the precision of global inversion. For example, minimizing reconstruction error doesn't necessarily need a low forward approximation error, while the consistency error integrates errors from both forward and inverse processes. Most works involving approximate one-to-one mappings focus on inverse accuracy alone, such as in image classification and recovery.

I will keep my score

We appreciate the reviewer’s time and efforts, and we apologize for the delay in responding to your questions about the short presentation of MetaInv. Given the importance of the switching algorithm, which is included in the title as an essential contribution, we wanted to take the time to carefully address your concerns and revise the manuscript accordingly. Below, we outline the key motivations for this work towards the hybrid method, followed by an analysis of the specific difficulties addressed during the algorithm development.

This study focuses on a specific class of inverse problems requiring deterministic two-way mappings for tasks such as physical system state estimation, image classification and recovery, and adaptive control. Since the objective is to recover point estimates with forward-inverse consistency, it necessitates models with explicit invertibility.

As the reviewer recognized, we have targeted two representative invertible architectures and presented the analysis both theoretically and empirically. Then, we found that, even with a narrowed scope, many factors and scenarios need to be considered when solving specific inverse problems. The differences can be evident when considering:

• Data types and inverse learning tasks: Even using the same data, there can be significantly different tasks. For example, image datasets (e.g., MNIST and CIFAR10), besides the major group of density estimation, also have image classification and recovery tasks. The latter emphasizes forward classification accuracy and inverse reconstruction. Similarly, physical systems involve both stochastic modeling to quantify uncertainties and deterministic inverse/two-way mappings to estimate precise system states.

• Application-specific factors: Beyond data type, specific characteristics of application cases play a critical role in model performance. These include but are not limited to:

  • i) The complexity of the underlying mapping - nonlinearity, dimension, coupling of variables, chaotic correlation, etc. Theorems 1 and 2 analyze the limitations of i-ResNet.

  • ii) The information redundancy of inputs - if the data have sparsity to be compressed and if the information can be retained to trace back to inputs. For example, images usually have redundancy, while variables with physical meanings are independent with minimum redundancy. We have Theorem 3 and test results (Fig. 3, Fig. 9, Fig. 12, and Table II) using different convolution kernels.

  • iii) The precision requirements for forward and inverse predictions, which are quantified by three metrics in Sec. 3.1.

These application-specific factors, combined with the trade-offs of invertible architectures, motivate the idea of switching algorithms. We encountered difficulties in balancing the objectives of inverse learning and dynamically adapting to task-specific characteristics. Therefore, we designed the trust-weighted switching mechanism, a hybrid approach inspired by learning-augmented online control.

From (i)–(iii), the objectives of inverse learning, such as high expressive power, strict invertibility, low redundancy, bi-directional consistency, and computational efficiency, are often in conflict. For example, i-ResNet’s flexibility enables it to model complex mappings but may sacrifice precision in inverse mappings. DipDNN’s analytical invertibility minimizes inverse error but limits flexibility, particularly for tasks involving data redundancy.

To address these challenges, the switching mechanism incorporates the trust-weight $\beta$, which integrates forward accuracy, inverse accuracy, inverse consistency, and computational cost into a single evaluation function: $V_{\text{model}} = J_{\text{model, total}} + \lambda C_{\text{model}}.$

The evaluation of these factors is not fixed during learning. To ensure smooth switching and avoid trivial convergences, $\beta$ is updated dynamically: $\beta_{t+1} = \beta_t + \eta_t (V_{\text{i-ResNet}} - V_{\text{DipDNN}}),$ where $\eta_t$ is the learning rate (empirically set to 0.01 for stability); additionally, a parameter $J_{\text{threshold}}$ sets an acceptable performance range, ensuring robustness to data uncertainties such as noise. This parameter is crucial, as inverse learning is highly sensitive to such uncertainties.

The following revisions are made:

The MetaInv algorithm was included in the Appendix due to the space limit. We have moved it back to the main part and reduced the less important contents.

We revised the corresponding description of MetaInv in Sec. 4.2, with a special focus on the difficulties and contributions of the switching algorithm, as highlighted by the reviewer.

The details of implementing the algorithm are included in the Appendix, e.g., parameter setups.