Thank you very much for spending your time on carefully reviewing our paper. We really appreciate all the advice you provide to improve our paper. Please find below answers to your questions. We also summarized the comparison of the time and space complexity of our models in an independent thread.

It would be interesting of the authors could provide experiments with both their architectures with respect to computational cost, and highlight time of training etc.

Thank you very much for this advice. Since a few reviewers were also interested in this topic, both theoretical and experimental discussions are summarized in the global rebuttal. We would appreciate it if you could verify this. This will be added in the updated version of our paper. Please note that we did not directly compare training times in seconds but instead compared the number of floating-point operations (FLOPs) for each architecture, as training time is heavily dependent on factors such as the machine, code, and libraries used.

maybe an experiment with a more larger dataset than the current ones use in the paper.

We conducted an additional experiment using the convolutional version of our model (BLNNconv) on the problem of uncertainty estimation (as in Subsection 4.2) with the CIFAR-10 vs. SVHN dataset to illustrate the scalability of our method. For this problem, we implemented the model so that we first process the data through BLNNconv and then transfer it to the DUQ. The result compared to DUQ is as below. Our model is not only scalable to large-scale networks but also improves out-of-detection performance (the AUROC of SVHN). Using BLNNconv instead of the fully connected BLNN also improved the computation time (e.g., 2.5 times faster for the 5 first iterations).

We hope we addressed all your questions and concerns. We will be glad to provide further explanation and clarification if necessary.

Thank you very much for considering our rebuttal! We greatly appreciate your positive position on the contributions of our paper.

Thank you very much for spending your time on carefully reviewing our paper. We really appreciate all the advice you provide to improve our paper. Please find below answers to your questions. We also summarized the comparison of the time and space complexity of our models in an independent thread.

(W=Weakness, Q=Question)

W1+Q2

We apologize for omitting this important point. Please find in the global rebuttal both theoretical and experimental discussions on this topic. This will be added in the updated paper.

W2+Q1

The main goal of our work was to create a new paradigm for the bi-Lipschitz control and to analyse its behavior. Nevertheless, we understand this is an important direction of exploration. We conducted an additional experiment using the convolutional version of BLNN (BLNNconv) on the problem of uncertainty estimation (as in Subsection 4.2) with the CIFAR-10 vs. SVHN dataset to illustrate the scalability of our method. We implemented the model so that we first process the data through BLNNconv and then transfer it to the DUQ. The result compared to DUQ is as below. Our model is not only scalable to large-scale networks but also improves out-of-detection performance (the AUROC of SVHN). Using BLNNconv instead of the fully connected BLNN also improved the computation time (e.g., 2.5 times faster for the 5 first iterations). Moreover, the amortization method of Reviewer u3yR may be a fundamental solution to the computational complexity of our model (which can already at least scale to CIFAR10) but out of the scope for this current work.

W3+Q3

Thank you very much for this advice. Here, we would like to focus on the inverse Lipschitz and Lipschitz hyperparameters ($\alpha$ and $\beta$). Note that other hyperparameters such as batchsize, depth…, follow the basic properties of the core neural network ICNN, and therefore, not directly related to the goal of this work (see e.g., Schalbetter, Adrian. Input convex neural networks for energy optimization in an occupied apartment. MS thesis. ETH Zurich, 2020.).

We would like to start the discussion by reminding that there is already an analysis with different $\alpha$ and $\beta$ for uncertainty estimation in the Appendix Table 5 p.50. We discuss in detail how the value of $\alpha$ and $\beta$ influence the performance of the model. The influence is rather intuitive, and we would like to verify this with additional experiments with a downscaled FashionMNIST for further clarity. The observations are as follows:

1. Changes in performance due to hyperparameter changes are continuous (no hyper-sensitivity). | $\alpha$ | $\beta$ | Accuracy | Loss| AUROC MNIST| AUROC NotMNIST| |-|-|-|-|-|-| | 0.2 | 0.99 | 0.8600|0.08|0.84|0.95| | 0.2|1.0| 0.8630|0.08|0.84|0.95|

2. An increase of the domain of search leads to better performance due to higher expressivity. | $\alpha$ | $\beta$ | Accuracy | Loss| |-|-|-|-| |0.0|0.1| 0.5680|0.24| |0.0|0.2 | 0.7485|0.14| |0.0|0.3 |0.7825|0.11|

3. But too loose smoothing leads to worse performance. | $\alpha$ | $\beta$ | Accuracy | Loss| AUROC MNIST| AUROC NotMNIST| |-|-|-|-|-|-| |1.0|4.0| 0.1070|0.10|0.50|0.50| |1.0|1.0| 0.8455|0.11|0.81|0.98|

4. Too low sensitivity leads to worse out-of-distribution detection. | $\alpha$ | $\beta$ | AUROC NotMNIST| |-|-|-| |0.0|0.2 |0.42| |0.3|0.2 |0.83|

Moreover, in Appendix G.3 we also investigated the influence of Lipschitz constant on convergence speed in many different Lipschitz architectures (Figure 18 p.50). While increasing the Lipschitz constant (i.e., decreasing the smoothness of the network) leads to slower convergence, this decrease in speed is the smallest for ours among the compared models. In that sense, our method is more stable with respect to these hyperparameters.

W4+Q4

Concerning the first part of W4, our theoretical guarantee is almost valid for usual practical scenarios. For example, the setting of Theorem 3.5 agrees with our experiments.

As for the potential extensions to other network architectures, we would like to first remind that our work is primarily foundational, providing a novel paradigm for the creation of bi-Lipschitz architectures and addressing some non-negligeable issues of the field. Therefore, this paper focuses on the fundamental design and properties of our model related to the control of bi-Lipschitzness and its theoretical analysis. We acknowledge that there are some applications outside the bi-Lipschitz control framework that we could not investigate. That is also why we have tried to provide an extensive discussion and analyses in the appendix and kept the formulation as general as possible to promote further extensions and other interpretations for future work.

Now, in the context of DEQ and implicit models, the LFT can be indeed re-formulated as finding the solution $z$ of $z=x-\nabla F(z) +z$ which corresponds to a DEQ. In that sense, our model can be regarded as a bi-Lipschitz DEQ. We believe this novel interpretation is interesting for future work to increase the generality of our model.

Furthermore, as mentioned in the first answer, we can extend our BLNN to convolutional layers as well. We just have to change the core neural network of the BLNN, which is the ICNN, to the convolutional ICNN proposed in the original work of Amos et al. (2017).

It seems that […] discussion on potential negative societal impacts […] is still missing.

We will ensure that a more detailed discussion is included in the revised version.

We hope we addressed all your questions and concerns. We will be glad to provide further explanation and clarification if necessary. Moreover, clarity is one of our major concerns. If you have any recommendations or some parts of the paper that were difficult to understand, we will be glad to improve them for the next version.

Thank you very much for spending your time on carefully reviewing our paper. We really appreciate all the questions highlighting the significant potential and future directions of our work. Please find below answers to your questions and some clarifications to other important points of your review.

(W=Weakness, Q=Question)

Q4

Yes, we can extend our BLNN to convolutional layers. We just have to change the core neural network of the BLNN, which is the ICNN, to the convolutional ICNN proposed in the original work of Amos et al. (2017). We will definitely clarify this in the revised version. In the next answer, we provide experimental results with this architecture.

W1

We conducted an additional experiment using the convolutional version of BLNN (BLNNconv) on the problem of uncertainty estimation (as in Subsection 4.2) with the CIFAR-10 vs. SVHN dataset to illustrate the scalability of our method. For this problem, we implemented the model so that we first process the data through BLNNconv and then transfer it to the DUQ. The result compared to DUQ is as below. Our model is not only scalable to large-scale networks but also improves out-of-detection performance (the AUROC of SVHN). Using BLNNconv instead of the fully connected BLNN also improved the computation time (e.g., 2.5 times faster for the 5 first iterations). Please also refer to the global rebuttal for this topic.

W2

This is indeed a limitation of our algorithm but also an improvement compared to prior works in some aspects. From the theoretical perspective, several (bi-)Lipschitz layers do not provide a provable approximation class, and, from the practical perspective, some bi-Lipschitz models are also restricted to strongly monotone functions (e.g., BiLipNet). We would like to add that most existing layer-wise approaches have difficulty in creating partially bi-Lipschitz networks (e.g., controlling the spectral norm of linear matrices cannot handle this case) but ours can easily realize this with the PBLNN. Finally, while out of the scope of this work, there are some straightforward ways to increase the expressive power of BLNN. Please see the next answer.

Q1

The following answer is also related to W1 and W2, and is therefore a bit longer.

Yes, the vanilla BLNN is necessarily a strongly monotone function. However, our model still has a higher expressive power and tighter bounds than other (bi-)Lipschitz units (= the most basic architecture guaranteed to be bi-Lipschitz without using composition of several bi-Lipschitz entities). Indeed, most existing models can only create simple (bi-)Lipschitz units with low expressive power (e.g., only linear) and they have to compose those units to achieve higher expressive power, but this leads to looser bounds. In comparison, our model can with only one unit achieve high expressive power and keep tight bounds. This is an improvement realized thanks to a parameterization not layer wise. Experiments of figures 2 and 3 were designed to illustrate this fact.

Now, we can of course, like any previous (bi-)Lipschitz methods, start to employ our BLNN unit as a component of a deeper network. For example, we can successively compose several BLNNs to improve its expressive power. With this approach, it can indeed represent non-monotone functions. We have tested to fit the sign function and the composition of 5 BLNNs achieves an accuracy around 0.

Some other extensions to increase the flexibility were already mentioned such as the PBLNN, and the BLNN used as a pre-processing stage for the next networks as we did in the answer to W1. Future research should work on how to use this novel unit in various fields inside and outside bi-Lipschitz problems.

Nevertheless, this direction is slightly out of the scope for this work. Indeed, our work is primarily foundational, providing a novel paradigm for the creation of bi-Lipschitz architectures and addressing some non-negligeable issues of the field. Therefore, this paper focuses on the fundamental design and properties of our model related to the control of bi-Lipschitzness and its theoretical analysis. One of the main contributions of this work resides thus in the construction of such a general framework for improved bi-Lipschitz control equipped with theoretical guarantees. While we ran several experiments related to bi-Lipschitz control to illustrate the benefits of our model, we acknowledge that there are some applications outside the bi-Lipschitz control framework that we could not investigate. This is also why we have tried to provide an extensive discussion and analyses in the appendix and kept the formulation as general as possible to promote further extensions and encourage other interpretations for future work.

Q2

Thank you for pointing out this interesting possibility. We also believe that BLNN might perform better in this setting by composing many BLNNs. However, this is out of the scope for this paper as explained in the above answer. This is still one of the first avenues we will investigate for future works.

Q3

Amortizing the LFT computation as the cited paper may be a fundamental solution to the computational complexity of our model (which can already at least scale to CIFAR10). However, this approach is currently out of the scope of this work as the main goal of our paper is to provide a bi-Lipschitz architecture with guaranteed bi-Lipschitz control and build its foundation starting with well-known basic algorithms such as the gradient descent. This amortization technique is more involved as we have to carefully analyse how such amortization influences the provable bi-Lipschitz bounds of the architecture, which is not trivial.

We hope we addressed all your questions and concerns. We will be glad to provide further explanation and clarification if necessary.