Response to Reviewer ndCL

Thank you for your review and recognizing the practicality of our solution. For your concern on the significance of the problem it addresses, we reply to all your comments below.

Weakness 1: Planar flow, the focus of this work, is one of the earliest methods introduced in the normalizing flow literature, and numerous advanced normalizing flows such as affine coupling flows, neural spline flows, etc., have been developed, showcasing superior performance. Given this context, the contribution of this work appears quite limited. To justify its contribution, it would be imperative to demonstrate that the modifications introduced significantly enhance the performance of planar flow. Ideally, it should now be on par with, or outperform, the more recent flow methods. This point leads to the second concern.

A1: We agree that the more recent flow methods are powerful in specific tasks. However, the expressiveness and scalability of the these flow methods come with a price — they are more intricately designed and require extra attention on parameter tuning/training, which can be inefficient in solving some variational inference tasks. This inefficiency is more severe when they are run on CPU-only devices.

While we agree that intricate models with fine-tuned parameters are powerful in specific tasks, we believe that simplicity can be a strength. Our objective was to provide a clear and accessible solution to general variational problems, where ease of implementation and computational efficiency are crucial, e.g., sampling from a customize distribution, posterior inference in Bayesian statistical regression, and low-dimensional VAE.

Weakness 2: The empirical experiments conducted in the study lack a comprehensive comparison with a variety of recent flow methods. Moreover, the paper only delves into variational inference applications, omitting performance evaluations on density estimation and the quality of synthetic data generation. To solidify the work's standing and importance, an expansion of the experimental section to include these aspects is necessary.

A2: Thank you for your suggestion. We have conducted an extra comparison with a recent flow methods, neural spline flows (NSFs). The updated comparison reaffirms the parameter efficiency of the new planar flows. We have included the results of the NSFs experiment in Figure 6, and a table summarizing the parameter counts is now available in Appendix D in the updated version.

As indicated by our title, Variational Inference with Singularity-Free Planar Flows, our focus is solely on variational inference. One reason for this focus is that the inversion of the planar flow does not have a closed form, making it less attractive for density estimation and data generation.

Response to Reviewer seL3

Thank you for your thorough review and positive feedback. Your acknowledgment of the importance and the effectiveness of our proposed methods is encouraging.

We reply to all your comments below.

Weakness 1: No theoretically justification why the new parameterization that removes the singularity would makes the approximation better.

A1: We understand your concern. While we don't have a formal theoretical proof, we believe our approach is logically grounded. We would like to highlight the logical coherence in our method, which, while not a formal theoretical proof, serves as a rational basis for the approximation improvement.

The training instability and suboptimal convergence arising from the singularity are predominantly numerical issues. The presence of an unstable gradient around the singularity renders certain distributions within the variational family practically unattainable. Consequently, this limitation restricts the expressiveness of planar flows. By eliminating the singularity, we can explore all distributions during training, which leads to better approximation in general.

Weakness 2: The experiments are all in relatively low dimensional space (the VAE example uss a latent dimension D=20). It is not clear if the benefit of the proposed method would extend to higher dimension problems.

A2: We appreciate your insight into the potential limitations of low-dimensional experiments. We took your suggestion and run the VAE experiment with a higher dimension $D=40$. Similar results are observed. This consistency supports the robustness of our findings, suggesting that the observations are not merely circumstantial but hold in higher-dimensional spaces as well. We have included this experiment in Appendix G.

Note that we did not run the IAF and NSF for $D=40$ as changing to a higher dimension necessitates a corresponding adjustment in the neural network size of IAF and NSF. Unfortunately, we did not have sufficient time to thoroughly test the neural network size to ensure comparable performance.

Response to Reviewer KdQZ

Thank you for your thorough and insightful comments. We are happy to see that you find our method useful and our paper well-written.

We reply to all the points below.

Weakness 1: The particular choice of the $m(\cdot)$ function seems a bit arbitrary. As argued in the paper, all we need is when $m(x) = \mathcal{O}(x)$ when $x \rightarrow 0$. There are many functions that satisfy this condition. To confirm that the issue of the original parameterization is indeed the singularity, the authors should pick a few other $m(\cdot)$ functions satisfying $m(x) = \mathcal{O}(x)$ and check if they share similar performance.

Question 2: Have the authors tried other function forms of $m(\cdot)$ that satisfy the condition $m(x) = \mathcal{O}(x)$?

A1: We regret that we did not make it clear how we chose the function $m(\cdot)$. The sufficient condition we stated was in the context of removing the singularity. There are other implicit conditions as well. The main role of $m(\cdot)$ is to squeeze the unconstrained dot product to the required region $(-1, \infty)$, which in turn reparameterizes the unconstrained $v'$ to the feasible values.

We chose $m(x)=x$ if $x \ge 0$ and a simple continuously differentiable extension to $x<0$, so that a minimal reparameterization is imposed on the unconstrained $v'$. This choice aligns with our belief in simplicity and minimizes the computation required.

Certainly, there are other more complicate functions satisfying all the conditions. However, they might just introduce unnecessary computation. We appreciate the reviewer for raising this question. We have modified the corresponding section to enhance clarity in the updated version (page 5).

Weakness 2: The idea is so simple that it is merely a parameterization trick. I am not sure if something similar has been done before. However, this may not be an actual weakness of this paper, as the reviewer is not particularly familiar with the literature on the planar flow.

A2: We agree that it is a parameterization trick. To the best of our knowledge, our work is the first attempt to address this specific issue. Frankly, this work is not groundbreaking. Our objective is to provide a clear and accessible solution to general variational problems, particularly in scenarios where ease of implementation and computational efficiency are crucial.

Weakness 3: One advantage of the planar flow in this paper is that its performance is comparable to the IAFs and SNFs but the planar flow has fewer parameters. One experiment to further strengthen this point is to compare the wall-clock running time of each flow.

A3: Thank you for your valuable suggestion. The planar flow is indeed faster and requires less computation resource than other flows due to its simplicity, especially with the minimal reparameterization we proposed. Based on our experiments, the new planar flows are approximately faster by a few hours than IAFs and SNFs for completing a replicate of the VAE experiment. We did not report this in the paper since the wall-clock running time we collected is not robust enough due to factors beyond our control.

We also note that the wall-clock running time is not the only metric and does not fully demonstrate the strength of the planar flow. For example, the planar flow is also trained on GPUs (to facilitate a fair control experiment), while it only requires small GPU usage, hence low power consumption. Additionally, it is possible to train planar flows efficiently on CPU-only devices.

Question 1: Have the authors generated images from the trained VAEs on the MNIST as a sanity check?

A4: Thank you for pointing it out. We have performed the sanity check as you suggested. The images generated from the trained VAEs are reasonable, which indicates that our models are properly trained. We have included the generated images in Appendix F.

Response to Reviewer Z1xs

We sincerely appreciate your progressive perspective on the nature of meaningful contributions to the literature. Your recognition that a paper doesn't necessarily need to be groundbreaking to make a significant impact resonates deeply with our intentions behind this work.

We reply to all your comment, suggestion, and questions below.

Weakness 1: The other perspective on the simplicity of the proposal is that it is not necessarily quite intellectually rich. I say this in the following sense - the related works points to many papers that choose different parameterizations for the coupling functions in normalizing flows. A pushback then is to say that changing a parameterization in the form they've done here is not that conceptually different from just proposing other transformations that don't have the original issue in the planar flows. The authors make a remark about the parameter efficiency of this method (which is true, it's a very simple function), but hopefully there'd be a bit more here.

A1: We acknowledge that our approach is intentionally simple, and we understand the concern about its conceptual depth. In choosing simplicity, our intention was to provide a clear and accessible solution to general variational problems, particularly in scenarios where ease of implementation and computational efficiency are crucial.

Suggestion 1: Some related work that the authors should include:
Other works that lay the groundwork for flow-based variational inference in scientific computing:

• Boltzman Generators, Frank Noé, Simon Olsson, Jonas Köhler, Hao Wu, 2019
• Flow-based generative models for Markov chain Monte Carlo in lattice field theory, Michaels S. Albergo, Gurtej Kanwar, Phiala E. Shanahan, 2019.

A2: Thank you for your valuable suggestion. We have diligently reviewed the papers you recommended, and incorporated references to these works in the updated version (page 2).

Question 1: Can the authors explain if their patch is relevant to the generalization of planar flows (Sylvester flows)?

A3: The Sylvester flow is considered as the generalization of the planar flow since it has the same form, $f(z) = z + A h(Bz + b)$, but allows for matrix weights. Our patch to the planar flow does not change the dimension of the weight parameters, so it is different from the approach of the Sylvester flow. However, there are similar constraints on the parameters of the Sylvester flow, $r_{ii} \tilde{r}_{ii} > -1$. Hence, it is possible to adapt our patch for the Sylvester flow to improve its expressivity/trainability.

Question 2: Can the authors justify their claim of parameter efficiency by providing parameter counts and pushing e.g. the number of parameters in the neural networks in the neural spline flows down to their minimum to maintain performance? Splines are for example clearly more expressive, the question is just what is the minimal network size that makes them competitive with the proposal here. This doesn't seem like that hard of an experiment to run either, but of course I am sympathetic to the overhead of more experimentation.

A4: Thank you for your suggestion regarding the experimental design for neural spline flows (NSFs). Your experimental design for NSFs aligns with our treatment to IAFs. We promptly conducted this experiment; however, it's worth noting that the training for NSFs required a considerably longer duration compared to other flow methods. We apologize for the delayed response.

The updated comparison reaffirms the parameter efficiency of the new planar flows. We have included the results of the NSFs experiment in Figure 6, and a table summarizing the parameter counts is now available in Appendix D in the updated version.

Question 3: Are there any higher dimensional experiments that the authors could run? The downside of low-d is it's hard to understand if the observations are generic or circumstantial.

A5: We appreciate your insight into the potential limitations of low-dimensional experiments. We took your suggestion and run the VAE experiment with a higher dimension $D=40$. Similar results are observed. This consistency supports the robustness of our findings, suggesting that the observations are not merely circumstantial but hold in higher-dimensional spaces as well. We have included this experiment in Appendix G.

Note that we did not run the IAF and NSF for $D=40$ as changing to a higher dimension necessitates a corresponding adjustment in the neural network size of IAF and NSF. Unfortunately, we did not have sufficient time to thoroughly test the neural network size to ensure comparable performance.