Adaptive Resolution Residual Networks

We appreciate your engagement with our work. We have provided a revised manuscript that includes more thourough experiments, that is more concise, and that better situates and differentiates our method relative to prior methods.

Q1.1) The application scenario of the algorithm is not clear. The proposed algorithm can only deal with the resolutions corresponding to those of Laplacian residuals.

A1.1) As you have noted, rediscretization applies in resolution steps that correspond to the resolutions of Laplacian residuals. While not ideal, we believe this is in fact an excellent compromise. Given an input whose resolution falls just between the resolution of two Laplacian residuals, the input can simply be rediscretized to the resolution of either residual block in a preprocessing step, which still results in excellent performance and improved computational efficiency, as shown in our experiments. This compromise is what allows us to make the resolution of layers nested within Laplacian residual blocks fixed while making the resolution of the whole network adaptive. This allows us to use standard layers, unlike previous methods that feature adaptive input resolution. This is a central aspect of the novelty of the method, and it makes it much easier to integrate with existing work on convolutional neural networks.

Q3) The paper does not compare with networks have adaptive input resolution.

A3) We do not see this as a detraction to the novelty of our method. ARRN is a drop-in technique that provides adaptive input resolution for residual networks built with standard layers. It makes most sense to compare ARRN with typical convolutional neural networks that are constructed with similar standard layers, as it reveals that our drop-in technique causes no loss in performance. This comparison cannot be made against existing methods with adaptive input resolution methods because they are not drop-in techniques. This is because they impose difficult design constraints on the layers they contain, which is explained in detail in the updated related works section. In contrast, our method is free from these limitations.

Q2) The method performs poorly without Laplacian dropout. It is not clear whether Laplacian dropout is responsible for improved robustness at low resolution.

A2) The effectiveness of Laplacian dropout is supported by our experiments. With Laplacian dropout, rediscretized ARRNs show overwhelming improvement in robustness at low resolution without compromise in performance at high resolution, even compared to well-established convolutional neural networks that specialise at a single high resolution. Without Laplacian dropout, rediscretized ARRNs show very poor robustness at low resolution. This poor performance is not an indication that the method is flawed, but an indication that Laplacian dropout is responsible for the improvement in robustness. This poor performance is without practical consequence because Laplacian dropout will be employed in every use case that aligns with the core motivation of the method, which is to allow adapting high resolution ARRNs to low resolution ARRNs for improved computational efficiency without re-training.

Q4) How are the filter kernels set?

A4) We have updated the implementation paragraph of subsection 4.1 to provide detailed parameters for the kernels.

Q1.2) The experiments only cover small resolution datasets.

A1.2) We have updated our experiments to cover the higher resolution TinyImageNet dataset, and also include a larger set of baseline methods spanning all usual variants of ResNet, WideResNetV2, MobileNetV3, and EfficientNetV2. Our method ARRN remains more robust at lower resolution than these standard methods while also beating their performance at the highest resolution. We would like to include the ImageNet dataset in our experiments however given limited time and computational resources we are not able to cover it appropriately. We believe our contribution is solid nonetheless as the novel properties of ARRNs are well supported by theory.

We hope that this comment, the improved manuscript, and the updated set of experiments address your concerns.

We appreciate your engagement with our work. We have provided a revised manuscript that includes more thourough experiments, that is more concise, and that better situates and differentiates our method relative to prior methods.

Q2) The novelty is limited. As said in the related works section, this work is merely an extension of Lai 2017 & Singh 2021.

A2) We maintain that Singh 2021 is unrelated to adaptive resolution approaches, and that ARRN is fundamentally distinct from Lai 2017. [Edited to discuss references separately] This prior method uses residuals chained by increasing resolution, and can adapt its output resolution by omitting residuals at the tail of the chain, which is well suited for super-resolution tasks. In contrast, ARRN uses residuals chained by decreasing resolution, and can adapt its input resolution by removing residuals at the head of the chain, which is more appropriate for classification tasks. Removing residuals at the head of the chain is nontrivially different from removing residuals at the tail of the chain, as later residuals must not be disturbed by the removal. We construct ARRNs around a novel theoretical analysis of this constraint.

We also highlight that prior methods which allow adaptive input resolution are either based on neural operators or implicit neural representations, which operate on unusual signal representations that are incompatible with most standard deep network layers. This covered in detail in the revised related works section of the paper. In contrast, ARRN uses the most ubiquitous form of signal representation, and is compatible with almost all standard deep learning layers. We believe this is key to the usefulness of our method to the community, as it allows easily creating adaptive resolution methods that can leverage a wide range of existing techniques.

Q1) The largest resolution in the experiments is 32 \times 32. The CIFAR10 dataset is not suitable to evaluate ARRN.

A1) We agree that evaluating methods across a larger space of resolutions is important. We have updated the experiments to cover the higher resolution TinyImageNet dataset, and also include a larger set of baseline methods spanning all usual variants of ResNet, WideResNetV2, MobileNetV3, and EfficientNetV2. Our method ARRN remains more robust at lower resolution than these standard methods while also beating their performance at the highest resolution. We would like to include the ImageNet dataset in our experiments however given limited time and computational resources we are not able to cover it appropriately. We believe our contribution is solid nonetheless as the novel properties of ARRNs are well supported by theory.

Q3) The inference time seems excessive. How does ARRN compare to prior works in this respect?

A3) The inference time of all methods is now included in Figure 6, as you requested. We apologize if it was not clear that we display the inference time not for a single batch, but for the sum of all batches across the whole dataset, excluding data loader overhead. While our implementation could be improved upon, ARRN achieves a reasonable inference time relative to prior methods that have had years of tuning from the community, and that are not burdened by the constraints imposed by adaptive resolution.

Let us know if these updates to not address your concerns.

We appreciate your engagement with our work. We have provided a revised manuscript that includes more thourough experiments, that is more concise, and that better situates and differentiates our method relative to prior methods.

Q3.1) The experiments are not sufficient.

A3.1) The updated experiments cover the higher resolution TinyImageNet dataset, and also include a larger set of baseline methods spanning all usual variants of ResNet, WideResNetV2, MobileNetV3, and EfficientNetV2. Our method ARRN remains more robust at lower resolution than these standard methods while also beating their performance at the highest resolution. We would like to include the ImageNet dataset in our experiments however given limited time and computational resources we are not able to cover it appropriately. We believe our contribution is solid nonetheless as the novel properties of ARRNs are well supported by theory.

Q3.2) Can ARRNs be used for other tasks such as super-resolution?

A3.2) We note that Lai 2017 proposes a form of inverse Laplacian residual block which has been highly successful at adaptive super-resolution tasks. [Edited to cite correct reference] Whereas our Laplacian residuals are ordered by decreasing resolution, and can adapt their input resolution by removing residuals at the head of the chain, inverse Laplacian residuals are ordered by increasing resolution, and can adapt their output resolution by omitting residuals at the tail of the chain. This inverse Laplacian residual block can in principle be combined with our Laplacian residual block to form U-Nets that map arbitrary resolution signals to arbitrary resolution signals, which may be useful in tasks such as segmentation and depth estimation.

Q1) Some related studies are necessary.

A1) We thank you for pointing our attention towards an important class of methods we had not considered in our related works section. We have added a discussion of implicit neural representations which overviews the concepts that are central to their formulation, and which highlights the way they differ from neural operator-based methods and our own method.

Q2) The authors highlight that it is impractical for the majority of deep learning methods to apply a single network at various signal resolutions. Could you provide some examples of such application scenarios in the paper?

A2) We have updated our introduction to better acknowledge methods that allow arbitrary resolution, and to give a concrete example for the impracticality of fixed resolution methods used in a variable resolution context. We are most interested in the scenario where a network trained at high resolution performs inference at low resolution.

We hope that this comment, the improved manuscript, and the updated set of experiments address your concerns.

We thank you for your engagement with our work and for the feedback you have provided.

Q1/Q2) The experiments feature datasets that have relatively small resolution and that could be more diverse. Improving this would make the results more convincing.

A1/A2) We have revised our work to include an improved set of experiments that cover the higher resolution TinyImageNet dataset, and that also include a larger set of baseline methods spanning all usual variants of ResNet, WideResNetV2, MobileNetV3, and EfficientNetV2. Our method ARRN remains more robust at lower resolution than these standard methods while also beating their performance at the highest resolution. We also include the inference time of all methods across different resolutions for transparency. While our implementation could be improved upon, ARRN achieves a reasonable inference time relative to prior methods that have had years of tuning from the community.

The revised version of our work is also edited to be more concise, and includes a more complete related works section that helps situate and differentiate our contribution relative to prior works.

We hope our comment answers your questions, and that you find the added experiments improve the quality of our work.