The Star Geometry of Critic-Based Regularizer Learning

We thank the reviewer for their detailed review and positive comments that our paper is "very well-written" and that our theoretical framework "opens up a number of new research directions both theoretically and numerically." Below we discuss some of the main questions that were raised.

How do we obtain uniqueness

Thank you for raising this point. The reason why Theorem 2.4 does not require an "almost everywhere" stipulation is that our optimization problem is over the class of star body gauges, as opposed to star-shaped gauges, or a more general function class. At a high-level, star body gauges have additional structure that allow for us to obtain uniqueness, which can be lost when one considers more general star-shaped gauges.

In more detail, star bodies are in one-to-one correspondence with their radial functions (or the reciprocal of their gauge) and are uniquely defined by them. As a result, dual mixed volumes also exactly specify star bodies, e.g., for star bodies $K,\tilde{K}, L,$ and $\tilde{L}$, we have $\tilde{V}(K,\tilde{K}) = \tilde{V}(L,\tilde{L})$ if and only if $K = L$ and $\tilde{K} = \tilde{L}$ (see Lutwak [52]). This property aids in establishing uniqueness in our main Theorem.

If we relax the requirement of being a star body to simply being star-shaped, we can lose such a property. In particular, if we consider gauges of star-shaped sets, it could be possible for the dual mixed volumes of different sets to be equal because two star-shaped gauges can be equal "almost everywhere". For example, consider the following two star-shaped sets in $\mathbb{R}^2$: $K = B_{\ell_2}$ is the unit $\ell_2$ ball, and $L = B_{\ell_2} \cup \set{(0,t) : t \geqslant 0}$. Both sets are star-shaped with respect to the origin, but $K$ is a star body while $L$ is not. This is because the ray $\set{t e_2 : t \geqslant 0}$ where $e_2 = (0,1)$ intersects the boundary of $L$ infinitely many times. The gauges of $K$ and $L$ are equal to $||x||_{\ell_2}$ for any $x$ outside of the set of measure zero $\set{(0,t) : t \geqslant 0}.$ This is because for any $(0,t)$ with $t > 0$, $||(0,t)||_K = |t|$ while $||(0,t)||_L = 0$.

Line 130

Our apologies, "this map" refers to the map $x \mapsto ||x||_K$. We will fix this in the manuscript.

Remark 2.8

We thank the reviewer for noting this. Our goal in this remark was to show that if the distributions do not satisfy the assumptions of the theorem, there is a way to reweight the objective so that the assumptions of the theorem are satisfied. By positive homogeneity of the gauge, this essentially amounts to reweighting/scaling one of the terms $\lambda\mathbb{E}_{\mathcal{D}_i}[||x||_K]$ for some $\lambda \geqslant 0$. We will update the discussion in the manuscript to make this more clear.

Line 50

Thank you for pointing this out. We intended to use the phrase "integrate information about the measurements" in a vague sense, meaning that the "bad" distribution $\mathcal{D}_n$ depends on $y$ in some way. We will update this phrasing to avoid this confusion.

Line 29

Thank you for making this point. Indeed, ill-posedness additionally encompasses a lack of a solution's existence and whether it varies continuously in the data. We will update this in the revised manuscript and cite Arridge et al's survey.

We thank the reviewer for their detailed review and positive comments that our theoretical framework is "rigorous and methodologically sound" and that our results "address the theoretical gaps in understanding how regularizers are learned." Below we discuss some of the main questions that were raised.

Additional context on the adversarial regularization framework

In the adversarial regularization framework of Lunz et al [51], the distributions $\mathcal{D}_r$ and $\mathcal{D}_n$ are user-specified distributions that are aimed to capture “real” or “good” data and “noisy” or “bad” data, respectively. The choice of these distributions depends on the specific application one is interested in. In the context of inverse problems, $\mathcal{D}_r$ is often chosen to be the distribution of images similar to the image we wish to reconstruct and $\mathcal{D}_n$ is the distribution of noisy or poor reconstructions. For example, suppose one is interested in solving an inverse problem $y = \mathcal{A}(x) + \eta$ where $\mathcal{A}$ is known and $\eta$ is drawn from some noise distribution $\mathcal{P}$. Suppose we have prior knowledge about $x$ that it is drawn from a distribution $\mathcal{D}_r$ and we have access to many samples from this distribution (e.g., we want to solve inpainting problems on human faces, and we have access to many samples from the CelebA dataset). One choice for $\mathcal{D}_n$ corresponds to the distribution of backprojected reconstructions, which corresponds to taking one's measurements and "naively" inverting them using the pseudoinverse of $\mathcal{A}$: $\hat{x} \sim \mathcal{D}_n$ if and only if $\hat{x} = \mathcal{A}^{\dagger} y$ where $y = \mathcal{A}(x) + \eta$, $x \sim \mathcal{D}_r$, and $\eta \sim \mathcal{P}$. These images will be corrupted by noise that depends on both the measurement matrix $\mathcal{A}$ but also the additive noise $\eta$.

This backprojected distribution is a good choice for the "bad" distribution for two reasons. One is that the distribution contains noise that is pertinent to the inverse problem (i.e., the samples depend on $\eta$ and $\mathcal{A}$). Another reason is that backprojected reconstructions are often used as the initial iterate for gradient-based methods to solve

$\min_{x \in \mathbb{R}^d} \frac{1}{2}|| y - \mathcal{A}(x)||_2^2 + \mathcal{R}(x).$ Hence, if the regularizer $\mathcal{R}(x)$ has been learned to assign very low likelihood to backprojected reconstructions, using them as initial iterates to a gradient-based algorithm would aid in the algorithm making good progress initially, as the regularizer would give gradients that would push the algorithm away from these poor, noisy solutions.

We will add this additional discussion as background on the adversarial regularization framework in an updated version of the manuscript.

The significance of Theorem 3.1

Thank you for raising this point. At a high level, Theorem 3.1 is significant and has implications for regularizer learning for three reasons:

• This theorem introduces and analyzes a novel critic-based loss function for learning regularizers. To our knowledge, the only other critic-based loss function that has been considered in the literature is the loss from the adversarial regularization framework of Lunz et al [51]. Introducing novel loss criteria to learn regularizers is broadly useful for the field of regularizer learning, and opens new avenues for future research, both from a theoretical and practical perspective.
• The theorem offers novel insights into the structure of regularizers one would learn from this loss. We give explicit expressions for their structure and are able to compare and contrast such regularizers with those found via the adversarial regularization framework of Lunz et al [51]. This can be seen by comparing the visual examples shown in Figures 2 and 4.
• Analyzing this loss brings new challenges and technical contributions for both the field of star geometry and regularizer learning. Please see our global comment for more details on the new technical challenges that this theorem addresses.

We thank the reviewer for their comments and questions. Below we would like to address concerns/questions that were raised.

Regarding novelty in relation to [50]

We appreciate the reviewer's concern regarding novelty in relation to [50]. Please see our global comment regarding the novelty and significance of our work. The following points summarize our work's impact and novelty:

• Our research introduces novel theory and a new framework for understanding critic-based regularizer learning, addressing a significant gap in existing theoretical foundations.
• We tackle several novel technical challenges inherent to the critic-based setting, distinguishing our work from [50].
• This work also demonstrates the applicability and utility of star geometry tools for regularizer learning and broader machine learning contexts.

We would also like to highlight that while our results and example look similar to those in [50], all results in the present work are novel. Moreover, all examples we visualize in the paper are different than those in [50]. The only example that is similar is the one shown in Figure 7 in the supplemental. Our experiment is different, however, since we are showing that this star body induces a weakly convex (squared) gauge, which was not explored in [50].

Typos and notation

Thank you for your catching those typos. We will update them in our revised submission. Yes, $[x,y]$ refers to the line segment between these two points in $\mathbb{R}^d$, i.e., $[x,y] := \set{(1-t)x + ty : t \in [0,1]}.$ We will add this definition in the manuscript.

We thank the reviewer for their detailed review and positive comments that our insights offer "an interesting theoretical contribution". Below we would like to address concerns/questions that were raised.

Regarding novelty in relation to [50]

We appreciate the reviewer's concern regarding novelty in relation to [50]. Please see our global comment above regarding the novelty and significance of our work. We would additionally like to note that the new proposed $\alpha$-divergence-based loss functions are meant to offer an alternative to the adversarial regularization framework that still falls under the umbrella of "critic-based" regularizer learning frameworks. While we did not provide experiments in this paper, we believe our theoretical contributions provide novel insights into unsupervised, critic-based regularizer learning, an area that has been underdeveloped theoretically, and offer convincing evidence that other critic-based losses are worth exploring both from a theoretical and practical perspective.

Adding an overview

Thank you for this suggestion. We agree that this would help with readability of the manuscript. We will add this in an updated version of the manuscript.

Regarding Proposition 4.3

Yes, we agree that requiring positive homogeneity of each layer limits the activation functions one can use. However, such activations are commonly employed in such tasks. For example, the experiments that were done in Lunz et al [51] and in subsequent applications also used a network of this form, i.e., a convolutional neural network with no biases and LeakyReLU activations.

Regarding the concern of a lack of injectivity, we first note that Prop 4.3 requires that each layer $f_i(\cdot)$ is injective and positively homogenous. Note that the composition of any two injective and positively homogenous functions is again injective and positively homogenous. To see this, consider two injective and positively homogenous functions $f$ and $g$. Then note that for any $\lambda \geqslant 0$ and $x$, $f(g(\lambda x)) = f(\lambda g(x)) = \lambda f(g(x))$. To see injectivity, note that $g(x) = g(y)$ implies $x = y$. The same holds true for $f$. Hence if $f(g(x)) = f(g(y))$, then $g(x) = g(y)$ by injectivity of $f$. But by injectivity of $g$, $x = y$. Hence $f(g(\cdot))$ is injective.

When applying this to neural networks, note that our result requires each layer $f_i(\cdot)$ to be injective and positively homogenous. This means that we require that the linear layer $W_i$ composed with the activation function $\sigma_i$ must be injective and positively homogenous, i.e., $f_i(\cdot) = \sigma_i(W\cdot)$ must be injective and positively homogenous. This is automatically satisfied if the linear layer is injective and the activation is LeakyReLU, since it is bijective. This can fail, however, if one uses ReLU. For example, note that $\mathrm{ReLU}(I \cdot)$ is not injective, even though the identity matrix $I$ is invertible. Certain matrices $W$, however, allow for injective maps $x \mapsto \mathrm{ReLU}(Wx)$. See, for example, the notion of a "directed spanning set" in Puthawala et al or the injective 1x1 convolutions of Kothari et al.

Puthawala et al, Globally Injective ReLU Networks, Journal of Machine Learning Research 23 (2022) 1-55

Kothari et al, TRUMPETS: Injective Flows for Inference and Inverse Problems, Uncertainty in Artificial Intelligence (2021)

I would like to thank the authors for the even more extensive numerical comparison. While I would not consider MNIST to be a good example dataset, it is nonetheless more informative than currently provided experiments. I would like to emphasise two particular points which are a bit problematic in the comparison above. In my understanding, in [51] $\lambda$ can be chosen in the form provided, as the regulariser is assumed to be 1-lipshitz. However, by turning to star-bodies I do not believe you have that property, right? This certainly becomes more problematic when considering Hellinger based distances, as you have seen in practice - you need a very different regularisation parameter. As such, above seems like a very fine-tuned comparison, not actually representative of performance, as the Adversarial MSE can likely be fine-tuned to be even better performing by tuning that hyperparameter. I would also suggest including a TV or l1 regulariser to show comparison to model-based regularisation as some sort of baseline.

All in all however, I find this very encouraging, and if the authors agree with the points above and are happy to add these experiments to the paper with a discussion of the points above regarding parameter choice, I would be happy to increase my score.