Implicit Neural Representations and the Algebra of Complex Wavelets

5. How could the theory and method be used for image compression?

This is a good observation. Indeed, the WMM initialization scheme allows for an INR to express the singular information of an image better using a fixed number of neurons, since it doesn't rely on a random initialization to hopefully cover the singular parts of the image. We think that developing this into a more complete image compression method warrants a look into the training dynamics of INRs from a harmonic analysis perspective, which is a great direction for future work.

And of course, many methods for meta-learning from datasets, pruning, and so on can be applied to generically compress the set of weights in a neural network, which applies to INRs as well.

2. I think the discussion in the context of wavelet literature could have been broader. For example, I did not see any discussions on the topic of Daubechies wavelets. Are Daubechies wavelets also progressive?

Daubechies wavelets, being real-valued in the usual case, are not progressive: we have added a comment on this in the updated paper. One can certainly convert any suitable wavelet to a progressive one by taking the Hilbert transform of the wavelet.

3. If the authors think Shearlets may have any potential here, providing a discussion might be useful. ... Could a shear matrix be potentially helpful in reducing the error because of its ability to extract anisotropic features?

This is an interesting comment, and is indeed something that we have thought about. Empirically, architectures like WIRE have been observed by their authors to yield atoms in the first layer that resemble curvelets:

Saragadam, Vishwanath, et al. "Wire: Wavelet implicit neural representations." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Seeing how the shear matrix of a shearlet transforms is used in conjunction with a parabolic scaling matrix to transform a "mother wavelet," INRs like the ones we study are capable of incorporating shearing transformations via the learning of the weights in the first layer of the network, without any explicit enforcing of the form of the matrix.

However, the theoretical properties of systems of shearlets/curvelets are difficult to reconcile with INRs in the way we study them, as said properties are often asymptotic properties of a specifically designed system. Implemented naively, INRs can certainly use shearlets/curvelets in the first layer, but the higher powers of those atoms will probably not obey the relevant scaling rules that make said systems have nice theoretical properties. This is an interesting point for future research on the topic though, so we have made a note in the updated conclusion about this.

4. On some relevant works to cite from Grattarola, et. al., and Shaham, et. al.

Indeed, thank you for pointing us to these references. We have included them in the survey section of the updated paper.

1. $\psi$ is a function from $R^d$ to $C$, but somehow $W^0r$ and $b^0$ are in $R^{F_1\times d}$.

Good point: the function $\psi$ is understood to act "row-wise" on the matrix in $R^{F_1\times d}$, so that it extends to a function $\psi:R^{F_1\times d}\to C^{F_1}$. We have added a comment to this effect in the updated paper.

2. The notation in Theorem 1 (now Lemma 1) is not clear. What is the definition of the Fourier transform for functions in $C_0^\infty(U)$? What does this stand for?

$C_0^\infty(U)$ is standard notation to denote the set of infinitely differentiable functions whose support is compact and contained in $U$. We have added a note clarifying this definition in the paper.

3. What is the product from $t=1$ to $t=F_1$?

The large asterisk denotes a convolution of the terms indexed by $t=1,...,F_1$. We have added a note clarifying this notation to the paper.

4. Does $\hat{\beta}_l$ depend on $W^l$ and $b^l$?

We see the confusion here; indeed, we should not have used the lowercase letter $\ell$ here in more than one way. We have changed the index notation to remove this ambiguity. Thank you for pointing this out.

5. Is $*$ a convolution, over which domain?

Yes, it is a convolution in the Fourier domain.

6. The argument in Section 3.2 regarding the support of the product term in Eq. 2 is not clear.

Based on comments of other reviewers, we have removed this section from the updated paper in order to make room for other additions to the paper. Thank you for pointing out the potential typo, though!

7. In Section 4.3, what does it mean to model a signal as a sum of a linear scaling INR and a nonlinear INR?

We have updated Section 4.3 with new comparative experiments to study the proposed architecture, along with a more precise explanation of what we mean by this. Thank you for suggesting this clarification, we feel that it has improved the presentation of this section of the paper.

6. Why is restricting our focus to polynomial activations beyond the first layer interesting to study?

Polynomial activations, although not usually used directly in neural network architectures, are reasonable ways to approach the harmonic analysis of more general neural network architectures. We show how polynomial activations can be extended to analytic (holomorphic) activations and beyond in Appendix B.1. For the case of real-valued networks, for instance, any continuous neural network can be uniformly approximated by polynomials over a compact domain by the Stone-Weierstrass Theorem, a fact leveraged in the study of ReLU networks in the paper

Mehmeti-Göpel, Christian HX Ali, David Hartmann, and Michael Wand. "Ringing ReLUs: Harmonic distortion analysis of nonlinear feedforward networks." International Conference on Learning Representations. 2020.

Of course, complex numbers complicate the choice of activation functions a bit in this case. For instance, the modulus function is not holomorphic, and thus can't be approximated by polynomials without using conjugation, and progressive functions are not closed under conjugation in general, so the interplay between Theorem 1 (now Lemma 1) and the algebra of progressive wavelets does not apply here. We have remarked upon this in the updated paper and in the appendix.

7. Why is the Fourier transform stated by considering multiplication by a smooth function?

This is done for two reasons: the first is technical, the second is a matter of interpretation/emphasis.

The technical reason is to handle Fourier transforms of functions that are only locally integrable. ReLU functions, for instance, are only integrable locally, not globally, so their Fourier transform only exists in the sense of tempered distributions (i.e., via duality with the space of rapidly decaying smooth functions). Multiplying by a compactly supported smooth function assuages these issues, and can be used to reconstruct the Fourier transform of the whole signal if needed.

The interpretive reason is to emphasize the locality of the INR architecture. In the time-frequency analysis perspective that we take, it is important to understand wavelets and their powers as being localized in space. This has important interactions with the notion of a locally $\Gamma$-progressive function for some conic set $\Gamma$, since the singularities in a function might only have common direction locally. Take the indicator function of the unit disc in $R^2$, for instance. The singularities of the disc are located along the edges of the disc and point in the normal direction to the boundary. Because of this, the Fourier transform of this function has slow decay in all directions. However, if you localize to a small neighborhood of a point on the boundary of the disc via multiplication by a suitable smooth function, the Fourier transform will only exhibit slow decay in a small cone around the normal direction. This is where the notion of a conically progressive function becomes useful, as you can guarantee that all generated frequencies are in line with the direction of a desired singularity.

2. Eq (2) is just the Fourier transform of a slightly rearranged definition of the INR architecture they are studying.

This is a fair point, and perhaps we did not emphasize the correct aspects of the paper. The point of the work is not Theorem 1 on its own, but rather to see how it interacts with different choices of the template function, in particular template functions that constitute an algebra under multiplication. To shift the emphasis, we have "downgraded" Theorem 1 to a Lemma, and provided a comment remarking that it is really just an application of the convolution theorem.

3. How does the fact that the INR is composed of the sum of the template function's "integer harmonics" illustrate the expressivity of the INR?

This statement gives a rough bound on the type of function that can be expressed by the INR. That is to say, it dictates what functions can and cannot be represented by an INR. For instance, if the template functions in the first layer all have Fourier support in some cone (that is, they are $\Gamma$-progressive for some conic set $\Gamma$), then the INR will be unable to generate frequencies outside of $\Gamma$. Depending on what an INR user is looking for, this could be viewed as a helpful or harmful constraint, but one worth knowing about either way.

4. "The support of these scaled and shifted atoms is preserved so that the output at a given coordinate is dependent only upon the atoms in the first layer whose support contains..." -- How is this not a trivial statement? What is the relevance of this observation?

Indeed, this is a simple property of the considered INRs -- that statement was meant merely to state the last part of Theorem 1 (now Lemma 1) in words. The observation becomes relevant when we consider the later sections on the use of $\Gamma$-progressive functions, as it is unreasonable to expect a multidimensional function (such as an image) to be globally approximated by a $\Gamma$-progressive function for some nontrivial conic set $\Gamma$, but it is more reasonable to expect this to hold locally in the image. See the above example of the unit disc.

5. What is the purpose of Section 3.2?

This is a fair critique -- based on your comments, as well as those of other reviewers asking for some additions to the paper, we have removed Section 3.2 from the updated manuscript to make room for new material.

I thank the authors for their elaborate response. I have skimmed through the updated version of the paper, spending more time on sections that I found unclear in the previous version. I am satisfied with the updated version, and am now happy to recommend its acceptance; I have increased my score to reflect this.

In my opinion, the authors have made their paper a great deal more accessible and also more interesting to the ICLR community by including more explanations and experiments on Kodak.

As a final point, I find that the authors' explanation for the relevance of Lemma 1 in the rebuttal is still clearer than the one in the paper. Hence, I think the authors could further strengthen the paper by incorporating more of the points they mention in the rebuttal into the camera-ready version.

3. What is an obvious advantage of this particular implicit neural representation over a sparse continuous wavelet transform with a sufficiently expressive bank of filters (e.g., some base filters and some of their polynomial powers)? In particular, if one needs to use wavelet modulus maxima to initialize the representation. Could these advantages be concisely stated?

This is a very good point: we have added in more comparative experiments to Section 4 of the paper, including a comparison to a "sparse continuous wavelet transform," which we treat as an INR with no hidden layers beyond the template functions. We designed the sparse CWT INR to have a similar number of parameters to the architecture that we suggest, and observe that it attains a higher MSE on the test signal.

Beyond an empirical demonstration, there are a few reasons why an INR with polynomial/analytic activations may be preferred over a bank of filters as described. By coupling the wavelet atoms and their powers via the hidden layers, an INR can generate many coefficients that are correlated to each other in order to resolve a singularity. While a standard wavelet transform, or one using powers of wavelets as well, can also do this by careful selection of the scales and abscissa (this is precisely the statement of Lemma 1!), decoupling these coefficients means that each one needs to be considered independently. Of course, the coupling across scales reduces the degrees of freedom of an INR with hidden layers compared to a large bank of wavelets, but the way in which this coupling is done perhaps introduces an implicit bias that makes learning easier.

4. Could some numerical comparisons be done to place the analyzed method within context of other INRs?

Yes, we have added in more numerical experiments with a variety of architectural choices. See the updated Section 4.3 for more comparisons of using scaling networks vs not (that is, testing the "split" architecture) in both the regimes of real and complex wavelets. This has led to a new observation that makes sense in hindsight, but was not obvious from the results of the paper: when using a split architecture, complex wavelets perform better, but when not using a network to approximate the signal with scaling functions, it is preferable to use real wavelets. This is due to the fact that powers of progressive wavelets do not generate low-frequency signal content, while power of real-valued wavelets are capable of doing so.

We have also added in new experiments demonstrating the proposed approach on a dataset of images, also comparing to other choices of nonlinearities that have been used in INRs.

5. Is it possible to show the smooth parts and the wavelet parts in the low-pass+progressive wavelet decomposition?

We have pictured this in Figure 4 of the paper for the 1D test signal. As expected, the scaling network yields a smooth signal, while the wavelet network is close to zero apart from the singularities. In addition to showing how the low-pass and high-pass parts are captured respectively by the scaling and wavelet networks, we also show in the bottom row how the nonlinearities resolve the high-pass features from a set of relatively smooth template functions via the hidden nonlinearities.

1. The provable statements in the paper are not in any way non-obvious. Theorem 1 is a direct consequence of the Fourier convolution theorem.

This is a fair point; indeed, all that Theorem 1 expresses is the Fourier transform of a polynomial of functions in terms of convolutions in the Fourier domain. The value in the particular expression comes from the later interaction with the notion of a progressive template function, such as a wavelet or complex exponential. We have changed the paper by "downgrading" Theorem 1 to a Lemma, with a comment acknowledging the simplicity of the result.

2. The setting in which these proofs work is highly impoverished with respect to the setting of actual interest, which is that of non-polynomial nonlinearities, such as the ReLU. The shifting of frequencies along a cone does not hold for these, and complicated ringing processes emerge...

We would first like to note that analytic activation functions, such as the sinusoidal activation used in SIREN, was shown in

Sitzmann, Vincent, et al. "Implicit neural representations with periodic activation functions." Advances in neural information processing systems 33 (2020): 7462-7473.

to in fact outperform ReLU activation functions. Although the Theorem (now, Lemma 1) was stated for finite-degree polynomials, there is a corresponding result for general analytic activation functions in Appendix B.1, which is referenced after the statement of Lemma 1. Moreover, for real-valued networks with ReLUs, the paper

Mehmeti-Göpel, Christian HX Ali, David Hartmann, and Michael Wand. "Ringing ReLUs: Harmonic distortion analysis of nonlinear feedforward networks." International Conference on Learning Representations. 2020.

studies the Fourier properties of ReLU networks by approximating the nonlinearities by polynomials, via the Stone-Weierstrass theorem. In our case, the nonlinearities need to act on complex numbers, so that polynomials (without conjugation) can only approximate holomorphic activation functions, excluding options such as the modulus. It is a reasonable critique, so we have added a comment addressing this in the relevant appendix of the paper.

In the linked paper

Mallat, Stéphane, Sixin Zhang, and Gaspar Rochette. "Phase harmonic correlations and convolutional neural networks." Information and Inference: A Journal of the IMA 9.3 (2020): 721-747.

they do study relationships between rectifiers (ReLUs) and complex wavelet coefficients, but this is done in a way so that the rectifier is only applied to the real part of a complex signal. The ultimate goal of their approach is to understand the phase of the wavelet transforms of real-valued signals after rectifiers are applied. This,as you note, doesn't quite fit with the type of statements made in our work, as it is concerned with activation functions applied to complex-valued signals directly.