Instant Video Models: Universal Adapters for Stabilizing Image-Based Networks

We thank the reviewer for their feedback and respond to their comments below.

Generalization across architectures. While we have not tried different architectures within a single task, we considered several architectures in our experiments: NAFNet for denoising, HDRNet for image enhancement, and Depth Anything v2 for depth estimation. Since the submission, we have also added results for segmentation using DeeplabV3; we will include these results in camera-ready. These architectures cover a range of designs and include both lightweight and heavyweight models. While we believe our results are sufficient to support the main claim of the paper regarding the generality of the proposed method, if the reviewer feels this is critical, we are open to evaluating another architecture for one of the existing tasks in the camera-ready version.

Tuning $\lambda$. The failure mode under extreme noise ($\sigma = 0.6$) is not related to difficulty in tuning $\lambda$ but rather a combination of architecture- and task-specific factors. We have provided detailed analysis in response to the suggestions from reviewers LzQr and pzRZ.

Overall we found tuning $\lambda$ to be straightforward, due in part to the guarantees provided by the oracle bound. In general, we found good results with $\lambda$ in the range 0.2-0.4. We did not find that the results were sensitive to fine adjustments of $\lambda$ - variations within the oracle bound give a smooth, well-behaved tradeoff between accuracy and stability.

Evaluation on other tasks. We have added results on semantic segmentation since the submission and will include these in camera-ready. We would also point out that denoising and low-light enhancement are two sides of the same coin (although the noise model may need to be adjusted to, e.g., Poisson in low light).

We thank the reviewer for their feedback and respond to their comments below.

Explanation of spatial fusion. We have included additional details in Section B.3 of the supplementary material, although we agree that the overall clarity of these explanations could be improved. We will revise and expand our description of spatial fusion in the camera-ready version.

Improvements to plots. We agree that these changes would improve the readability of the plots, and will incorporate them in camera-ready.

Potential bias by duration $\tau$. In our current experiments, we fix $\tau$ to a constant value during training, so there is no risk of longer sequences dominating the loss. However, for training setups where the video duration varies widely between sequences, it would be a good idea to introduce a duration-balancing factor $1 / \tau$ as suggested. We will add a note to Section 3 describing this consideration.

Reduction in PSNR. We have double-checked these numbers and this does not appear to be a transcription error or a bug. We have also confirmed that our method gives considerably higher subjective quality on test videos (as can be seen in the supplement video).

We conjecture that there are two things going on here. First, PSNR (averaged over frames in this case) does not capture the jarring perceptual effect of the dropped frames - so the high PSNR of the base model is somewhat deceptive. This points to the well-known limitations of PSNR as a video quality metric. We would point to the significant improvement in temporal stability as quantitative evidence of improved perceptual quality.

Second, there are hints of the spatial-fusion denoising failure mode here - PSNR is much higher (exceeding the baseline) on shorter sequences. See the response to reviewers LzQr and pzRZ for an explanation and a potential solution.

Figure 2 explanation. By "tilt toward" here we mean "slopes more gently toward." That is, deviations from the correct prediction are penalized less heavily if they are in the direction of better stability. We will revise this explanation in the camera-ready version.

We thank the reviewer for their feedback and respond to their comments below.

Performance degradation under extreme noise. This is a great question, also raised by reviewer pzRZ. We duplicate our response below to make the rebuttal self-contained.

We have examined these results more closely. When applying the spatial fusion method under extreme noise, we see some blurring/ghosting artifacts after enough time has passed. These artifacts can look like hard edges "bleeding out" into the surrounding regions. We believe this is related to the use of a spatial fusion stabilizer on the output layer. Without this stabilizer, the network output is biased toward the current input frame. However, adding a spatial fusion stabilizer to the output provides an independent mechanism for information to flow between pixels, thereby weakening this bias.

Because we only see these artifacts after some time has passed, the problem may be resolved by training on longer sequences. We ran a quick experiment to test this idea. We trained the spatial fusion stabilizer under extreme noise on sequences of length 8 (the default in our past experiments) and 16. For length 16, we reduced the total number of training iterations by half, to compensate for the greater number of frames in each iteration. We evaluated these models on the full validation set (containing very long sequences). Training with length 8 gave validation PSNR 22.70 and instability 11.04, whereas training with length 16 gave PSNR 23.80 and instability 10.16. We expect this trend of improvement to hold as we further increase the training sequence length.

We will incorporate this new experiment, sample results, and the above discussion in the supplement of our camera-ready version.

Comparison against video-specific baselines. Thanks for the suggestion - we agree that the paper would be strengthened by comparisons against video-specific baselines, especially for tasks like denoising. Although we do not have time to add these comparisons now during the rebuttal, we intend to add at least one such baseline for the camera-ready version.

We would also like to point out that our method is potentially complementary to many video-based models. Consider, for example, FastDVDNet, a video denoising model that maps five noisy input frames to one centrally-aligned denoised frame. To predict a denoised video, the model is applied in a sliding-temporal-window fashion to the input. Therefore, on each invocation of the model, the time offset of each output or activation value shifts forward by one frame. This mirrors the single-frame models we consider in the paper, where the activations also shift forward by one frame on each invocation. For other models which operate on distinct (non-overlapping) chunks of frames, our objective simply changes from "temporal consistency between frames" to "temporal consistency between chunks." The effectiveness of our method in this case would likely depend on the duration of the chunks - as chunks become longer the temporal correlation between them will tend to fall.

We thank the reviewer for their feedback and respond to their comments below.

Comparisons with consistent video enhancement. Per the reviewer's suggestion, we have set up the author-published code for Bonneel et al. and have run their method on the NFS Laplacian task. We are currently seeing severe artifacts in the output of this method; we believe this is due to a misconfiguration and will continue to work on this throughout the discussion period. We will provide these results if we are able to obtain a valid comparison.

There are several key conceptual distinctions between our method and these prior works. First, unlike consistent video enhancement methods, our approach does not require that the output is a natural video. This flexibility is evidenced by our results for other tasks, such as depth estimation. Since the submission we have also performed experiments for semantic segmentation, which we will include in the camera-ready version.

Second, we consider not only stability, but also robustness, which allows our method to work even when the input video contains various corruptions (see Figure 6, Table 1, and Section 6.3 of our main paper). For example, the method of Bonneel et al. determines the amount of temporal smoothness to enforce by looking at low-level input cues, meaning it cannot smooth the output correctly in situations where the input video contains unstable corruptions.

Compatibility with transformer architectures. Our method is indeed compatible with transformer architectures. When dealing with transformer layers, care must be taken to ensure that the controller output has the correct shape: a 1D list of tokens, rather than a 2D feature map. This can be achieved in several ways: by applying the backbone after an existing input tokenization, by adding a tokenization layer within the backbone, or by tokenizing the output of the controllers. In all of these cases, the controller layers after the tokenization should be switched from convolutions to MLP layers or self-attention blocks.

We thank the reviewer for their feedback and respond to their comments below.

Lack of error bars. We agree that adding error bars would strengthen the experimental results. Before camera-ready, we will run one set of experiments (a particular model and task configuration) multiple times to generate confidence bounds.

Analysis of failure cases with spatial fusion. This is a great question, also raised by reviewer LzQr. We duplicate our response below to make the rebuttal self-contained.

We have examined these results more closely. When applying the spatial fusion method under extreme noise, we see some blurring/ghosting artifacts after enough time has passed. These artifacts can look like hard edges "bleeding out" into the surrounding regions. We believe this is related to the use of a spatial fusion stabilizer on the output layer. Without this stabilizer, the network output is biased toward the current input frame. However, adding a spatial fusion stabilizer to the output provides an independent mechanism for information to flow between pixels, thereby weakening this bias.

Because we only see these artifacts after some time has passed, the problem may be resolved by training on longer sequences. We ran a quick experiment to test this idea. We trained the spatial fusion stabilizer under extreme noise on sequences of length 8 (the default in our past experiments) and 16. For length 16, we reduced the total number of training iterations by half, to compensate for the greater number of frames in each iteration. We evaluated these models on the full validation set (containing very long sequences). Training with length 8 gave validation PSNR 22.70 and instability 11.04, whereas training with length 16 gave PSNR 23.80 and instability 10.16. We expect this trend of improvement to hold as we further increase the training sequence length.

We will incorporate this new experiment, sample results, and the above discussion in the supplement of our camera-ready version.

Smoothing of the noise residual. To answer this question, we look at the "learned EMA" configuration, which is the same as the "internal EMA" variant except that we train the decay parameters (with one parameter per feature channel). We consider the NAFNet NFS model trained under moderate noise with $\lambda = 0.2$. The table below shows the mean of the decay logits for each layer, along with the feature dimension to give a sense for the level of visual abstraction. Larger logit values correspond to beta near one (minimal smoothing). The last row is for the stabilizer on the model output.

By far, the most aggressive stabilization is applied to the model output. Within the residual-prediction backbone, we see the most stabilization in the intermediate layers. These intermediate layers correspond to higher-level features that should be more stable than either the noisy input image or the residual prediction. Note that although the mean logit values are quite high, there is significant variation across channels, with many channels having values in the 0-1 range. This suggests that some channels are much more amenable to stabilization, and supports the use of training over hand-tuning the stabilizer decay parameters.

We will include this new experiment and our discussion in the supplement.

Testing against other adversarial attacks. This is a good suggestion, although so far we have not tried this. In general, we would expect non-gradient-based attacks to be weaker than gradient-based attacks, and therefore easier to defend.