DVI:A Derivative-based Vision Network for INR

Thank you for your thorough and constructive feedback. We address your questions point by point below:

Q1: Clarify the relation with the derivations presented in (Xiao et al., 2023)

We confirm that we use the recursive formula for high order derivatives in (Xiao et al., 2023) to compute the derivative map at an accelerated speed. We will follow your suggestion to revise our claims throughout the paper as requested. We will also follow your suggestion to remove section 3.3 and use the space to increase the size of tables and figures.

Comments or suggestions:

• We will follow your suggestion to reword the claims about INR containing semantic information into sentences that more plainly and clearly state that high order derivatives from INRs are shown to be useful in solving tasks more effectively.
• We will follow your suggestion to tone down the claims regarding INSP's (Xu et al., 2022) inability to handle the tasks. We will also note in the main text that the comparison is greatly affected by this fundamental difference between the methodologies.
• We will follow your suggestion to move Appendix B.1 about normalization of the derivative maps into the main paper to make it integral to the methodology for how derivative maps are normalized.
• We will follow your suggestion to address the issues you identified in Line 064, Line 072, Lines 89-92, Line 153, as well as the problems in the Figures and tables.

We sincerely appreciate your thorough review and insightful questions, which will significantly improve the quality of our research!

Thank you for your thorough and constructive feedback. We address your questions point by point below:

Q1: Include diffusion-based methods for comparison

We included 5 diffusion-based methods for comparison across all vision tasks addressed in our paper. These include the 2 algorithms you suggested (StableSR [IJCV'24] and DiffBIR [ECCV'24]), as well as 3 additional algorithms (MedSegDiff-V2 [AAAI'24], VD-Diff [ECCV'24], and FlowDiffuser [CVPR'24]) to demonstrate the effectiveness of our method:

Table R1: Quantitative results (PSNR↑) for image super resolution task.

Table R2: Quantitative results (PSNR↑) for image denoising task.

Table R3: Quantitative results (DSC↑) for 3D volume segmentation task.

Table R4: Quantitative results (PSNR↑) for video deblurring task.

Table R5: Quantitative results (EPE↓) for video optical flow estimation task.

Our approach consistently improves performance across all tasks with minimal overhead: 1+ dB PSNR gain with only 1% cost for super resolution and 3% for denoising; nearly 10 percentage point DSC improvement for 3D segmentation with 7% overhead; and significant performance gains for video deblurring and optical flow with just 9-10% additional computation.

Q2: Provide the computational overhead

We provided the MAC (Multiply-Accumulate Operations) for each algorithm in Tables R1 to R5 in our previous response, as well as in Tables 1 to 4 in our initial submission. For your convenience, we rearranged typical comparison results below, showing both performance improvement and computational overhead for each comparison:

Table R6: Performance improvement and computational overhead on different vision tasks.

This demonstrates that our approach consistently improves performance across all vision tasks with minimal computational overhead, making it practical for real-world applications. We achieve this efficiency primarily by reducing the complexity of high-order derivative computations through our MLP-specific derivative computation paradigm. Furthermore, the overhead from our attention-based feature processing can be further reduced using PyTorch 2.2.2's FlashAttention-V2, making our method even more lightweight in future implementations.

We sincerely appreciate your thorough review and insightful questions. If our supplementary experiments and analyses have addressed your concerns, we would be grateful if you could consider adjusting your evaluation. If questions remain, we welcome further discussion to address any outstanding issues.

Thank you for your thorough and constructive feedback. We address your questions point by point below:

Q1: Discuss the computational overhead

We provided the MAC (Multiply-Accumulate Operations) for all algorithms in Tables 1-4 of our submission, including tests on more complex generative networks. For your convenience, we summarize key results below:

Table R1: Performance improvement and computational overhead.

Our approach improves performance with acceptable overhead across all tasks. For large networks (StableSR, EDSR, DiffBIR), the overhead is minimal (1.0%-9.7%). Smaller networks show higher relative increases but maintain reasonable absolute MAC counts. We achieve this efficiency through our MLP-specific derivative computation paradigm, ensuring scalability for larger networks where relative overhead becomes increasingly negligible. Future implementations could further increase scalability using PyTorch's FlashAttention-V2.

Q2: Why INR's high-order derivations help capturing semantic information

INR's high-order derivatives contain richer spectral information than raster representations, providing more comprehensive semantic information for visual tasks.Due to space limitations, we provide concise theoretical proofs for three visual tasks, with the remaining proofs being similarly derived.

Definition

Let $I\_d \in \mathbb{R}^{H \times W \times C}$ be a discrete image and $I\_h = \\{I\_{h\_0}, I\_{h\_1}^{1,0}, ..., I\_{h\_n}^{i,j}, ...\\}$ be the higher-order derivative map, where $I\_{h\_n}^{i,j}$ represents the sampled partial derivative $\frac{\partial^n I}{\partial x^i \partial y^j}$ with $i+j=n$. Here, $I$ denotes the continuous function represented by the INR.

Analysis: Super-Resolution

Proposition: In super-resolution, $I\_h$ contains richer semantic information than $I\_d$.

Justification: Defining semantic richness as: $\mathcal{S}\_{SR}(I) = \int\_{\\|\omega\\| > \omega\_0} |\hat{I}(\omega)|^2 d\omega,$ where $\omega_0$ is the Nyquist frequency in $I\_d$, we can derive: $\mathcal{S}\_{SR}(I\_h) = \sum\_{n=0}^{N} \sum\_{i+j=n} \int\_{\omega\_0 < \\|\omega\\| \leq \omega\_N} |\omega\_x|^{2i}|\omega\_y|^{2j}|\hat{I}(\omega)|^2 d\omega.$ Since $|\omega_x|^{2i}|\omega_y|^{2j} > 1$ for $\\|\omega\\| > \omega_0$ and $(i,j) \neq (0,0)$: $\mathcal{S}\_{SR}(I_h) > \int\_{\omega_0 < \\|\omega\\| \leq \omega_N} |\hat{I}\_d(\omega)|^2 d\omega = \mathcal{S}\_{SR}(I_d).$

Analysis: Denoising

Proposition: In denoising, $I\_h$ contains richer semantic information than $I\_d$.

Justification: Defining semantic richness as: $\mathcal{S}\_{DN}(I) = \frac{\int\_{\Omega\_S} |\hat{I}(\omega)|^2 d\omega}{\int\_{\Omega\_N} |\hat{I}(\omega)|^2 d\omega},$ where $\Omega\_S$ and $\Omega\_N$ represent signal and noise domains, and modeling noisy image as $I = I\_{clean} + \eta$.

Signal components exhibit directional coherence while noise is isotropic. For appropriately chosen derivatives: $\frac{\int\_{\Omega\_S} |\omega\_x|^{2i}|\omega\_y|^{2j}|\hat{I}\_{clean}(\omega)|^2 d\omega}{\int\_{\Omega\_N} |\omega\_x|^{2i}|\omega\_y|^{2j}|\hat{\eta}(\omega)|^2 d\omega} > \frac{\int\_{\Omega\_S} |\hat{I}\_{clean}(\omega)|^2 d\omega}{\int\_{\Omega\_N} |\hat{\eta}(\omega)|^2 d\omega}.$ Therefore, $\mathcal{S}\_{DN}(I\_h) > \mathcal{S}\_{DN}(I\_d)$.

Analysis: 3D Volume Segmentation

Proposition: In 3D volume segmentation, $I\_h$ contains richer semantic information than $I\_d$.

Justification: Defining semantic richness as: $\mathcal{S}\_{VS}(I) = \sum\_{k=1}^K \alpha\_k \cdot \int\_{\Omega\_k} |\hat{I}(\omega)|^2 \cdot H(\hat{I}|\_{\Omega\_k}) d\omega,$ where $H$ is entropy.

Since higher-order derivatives enhance:

1. Boundary contrast: $|\hat{I}\_h^{i,j,l}(\omega\_{boundary})|^2 \gg |\hat{I}\_d(\omega\_{boundary})|^2$,
2. Directional information: $H(\hat{I}\_h^{i,j,l}|\_{\Omega\_k}) > H(\hat{I}\_d|\_{\Omega\_k})$,

therefore, $\mathcal{S}\_{VS}(I\_h)>\mathcal{S}\_{VS}(I\_d)$.

We sincerely appreciate your thorough review and insightful questions. If our analyses and theoretical justifications have addressed your concerns, we would be grateful if you could consider adjusting your evaluation. If questions remain, we welcome further discussion to address any issues.