Space-Time Attention with Shifted Non-Local Search

• We hope the updates to Sections 3.1 and 3.2 better explain our method, and the meaning of “in-place computation”. By “in-place”, we simply mean no input video data is copied [wiki]. The indexing described in Section 3.2 is directly used within our code.

• We have updated the terms “reference/search” strides to “query/key” strides, respectively.

• We have updated Figure 7 in Section 4.1 to address your interest regarding how parameters (such as patch size and query stride) impact the search quality. We also gently note in the supplemental Section 6.2, an experiment uses a patch size of 7 and query stride of 4. Additionally, Sections 6.3 and 6.4 include ablations experiments regarding the key stride and space-time window sizes.

• Directly investigating the offsets, $\mathbf{F}$, is important. This is a motivation for Figure 3 in Section 3.3. However, directly comparing offsets to evaluate the quality of a search module will be misleading. This is because attention modules act as a kernel function on the input video. Thus different offsets may produce similar outputs, depending on the content of the value video ($\mathbf{V}$). Written agin, when the two offsets are not equal, $\mathbf{F}^{\text{Ideal}} \neq \mathbf{F}^{\text{Shifted-NLS}}$, the final output might be similar, $\mathbf{X}^{\text{out,Ideal}} \approx \mathbf{X}^{\text{out,Shifted-NLS}}$. It may also be true that when the two offsets are similar (but not equal), the final output might be dramatically different. This ambiguity is resolved by comparing offsets through their impact on the attention module's final output. The Video Alignment experiment in Section 4.1 provides an interpretable output target for the attention module. We thank you for pointing out this gap in our explanation and we have updated the text in Section 4.1 to clarify this point.

Not a Mixture. Our paper is not a mixture of the three listed papers.

• First, BasicVSR++ [1] is not considered in this paper as our method is inspired from non-local denoising methods, such as Video Non-Local Bayes [4].
• Second, RVRT [2] uses Predicted Offsets while our Shifted Non-Local Search outperforms this method by 3 - 7 dB on video alignment in Figures 6 and 7 (Sec 4.1). When replacing RVRT’s Predicted Offsets with our Shifted Non-Local Search, RVRT’s denoising quality improves by 0.30 dB in Table 1 (Sec 4.2).
• Third, IART [3] is an unpublished paper that provides less functionality than the Shifted Non-Local Search and requires 81 times the memory (relevant code link) when searching a window of size 9x9.

[4] Pablo Arias and Jean-Michel Morel. Video denoising via empirical bayesian estimation of space-time patches. Journal of Mathematical Imaging and Vision, 2018.

IART. IART [3] is an unpublished paper that is (1) computationally more expensive and (2) provides less functionality than this paper’s Shifted Non-Local Search. Firstly, IART requires a significant spike in memory consumption. When searching a spatial window of size $(W_s,W_s)$, IART must copy the input video $W_s^2$ times (relevant code link). For a 9x9 search (as in Section 4.1), IART requires the input video to be copied 81 times, while the in-place Shifted Non-Local requires no copying. Second, IART’s use of Pytorch’s grid_sample function restricts the method to use image patches of size 1, while our method uses image patches of size 3 and 7 in Sections 4.1 and 6.2, respectively. In the first and third rows of Figure 7, there is an approximate 3 - 4 dB difference between a patch size of 1 and 3.

RVRT uses Predicted Offsets. We agree that RVRT does indeed use an auxiliary network to predict offsets (named “Predicted Offset” in the paper) with optical flow and video features as inputs. In Table 1 from Section 4.2, we replace this small network with our Shifted Non-Local Search and observe the video denoising quality improve. We have updated the text in Section 4.2 to clarify this point.

Small Optical Flow Errors on the Sintel-Clean Benchmark. We have updated our text in Section 3.3 to clarify our point regarding the Sintel-Clean dataset. We agree with you: the results from the Sintel-Clean benchmark will be overly optimistic. Our discussion in Section 3.3 is to illustrate that even recent, large-scale deep networks still report an end-point-error of about 1 even on this “far from real-world” dataset. Yet, using OpenCV’s implementation of Farnback’s 2003 optical flow method (a computationally “cheap” method) combined with a large grid search of 41x41 pixels, the most similar pixels (i.e. the pixels used for attention) exist in a small radius surrounding the offset values. We further acknowledge this radius may grow as noise increases, which is a motivation for the Video Alignment experiment in Section 4.1.

Q1. Reproducing RVRT. One reason our reproduced RVRT network yields lower quantitative results may be due to our computational setup. We use a stand-by queue on SLURM, which requires training in 4-hour increments. While existing open-source packages claim to fully support this functionality, in reality, this may be unreliable [examples of previous issues: 1, 2]. Still, the STAN network is trained in the same fashion and yields significantly better results. Training time may also be a factor, which we discuss in more detail in Supplement Section 6.4.

Q2. Comparing STAN and RVRT. Since STAN processes multiple frames in parallel and RVRT does not, we are unable to directly compare the two networks. The updated text at the start of Section 4 hopefully clarifies this point. Notably, in Section 4.2 we do directly compare our proposed grid search against the auxiliary network used within the Guided Deformable Attention module (within RVRT). Table 1 shows the direct comparison between offsets computed from an auxiliary network (“Predicted Offsets” ) and our grid search (“Shifted Non-Local Search”). We also include ablation experiments for both RVRT and STAN in Sections 6.3 and 6.4, respectively.

I have read the responses and would like to keep my original evaluation.

Thank you for the comments on our presentation. We have updated Sections 3.1 and 3.2 to improve the readability of the method. In Sections 4.1 and 4.2, we have added more details to explain the intention of the experiments.

• [Sections 3.1 and 3.2]: We have changed our presentation to guide the reader through increasingly sophisticated search methods within attention modules. I.e. Global Attention -> Neighborhood Attention -> Non-Local Search -> Shifted Non-Local Search.

• [Section 4]: We have added a paragraph summarizing our experiment section.

• [Section 4.1]: We add motivation for using the Video Alignment task as our evaluation of the different search methods.

• [Section 4.2]: We add more detail to explain how RVRT’s auxiliary network (“Predicted Offsets”) are replaced with our grid search (“Shifted-NLS”).