Video Decomposition Prior: Editing Videos Layer by Layer

We thank the reviewer for taking their time out of a busy schedule and evaluating our submission. We are glad that reviewer 6Tks found our paper well-written. We greatly appreciate that reviewer 6Tks finds the test-time optimization factor to be a strength of our approach. We are happy to note that the reviewer found our approach performance better across different tasks. Before we address the specific weaknesses and questions of the reviewer, we highly recommend R-6Tks to please go through the webpage of video results, which was part of the supplementary.

Relighting Decomposition: We understand the reviewer’s concerns regarding the formulation of the relighting task. We would like to emphasize that the decomposition of a frame may not involve only additive decomposition($\alpha$-blending). What we explored in this work (one of our major novel contributions) is to perform multiplicative decomposition on frames of the video sequence. We derived one such multiplicative decomposition utilizing the gamma correction formulation for the frame relighting task. The interesting outcome of this decomposition is we are able to achieve SOTA performance on the relighting task without requiring any task-specific data. Not only that, we are also able to achieve high temporal coherence in the processed relit videos. Such qualitative comparison can be found in videos attached with the webpage(supp.).

Unified Framework: We hope to change the reviewer’s viewpoint about the VDP framework being artificial. We have taken great pains to figure out this method.

The core of our framework lies in its ability to adapt to different tasks while maintaining a consistent structure. Basically, each task consists of three integral components: prior knowledge (decomposition formulation), motion modeling (via $\alpha$-Net), and appearance modeling (via RGBnet). The consistent architecture of $\alpha$-Net and RGBnet establishes a unified way of extracting the motion and appearance features from a video, across different tasks. These intermediate motion and appearance features are then utilized with task-specific adaptations that are reflected in the decomposition formulations. Hence, the unifying factor across different tasks is the consistency in the architecture used to extract motion and appearance cues from the videos across all tasks.

Novelty Issues: We claim novelty on two terms. First, The relighting task formulation derived from gamma correction is novel for low-lit video enhancements and is unique to our work. Such formulation is not utilized in the work[1] mentioned as a reference by the reviewer.

Second, The VDP framework’s way of modeling motion and appearance in a video is unique to our work(utilizing $\alpha$-Net and RGBnet). Specifically, to our knowledge, no prior test time optimization technique has utilized FlowRGBs to model the motion of the video, and it is unique to our methodology.

With regards to the mentioned test-time methods[2,3], they are not at all designed for the VOS task but more for edit propagation tasks. These methods require per-frame masks obtained by image segmentation baselines. So, we do not see any overlap of these techniques with our framework apart from edit propagation tasks.

We also would like to emphasize that maintaining spatio-temporal coherence is a big issue when it comes to tasks like dehazing and relighting, as can be seen from the qualitative results provided in the webpage(supplementary material). Our novelty also stems from the fact that our method can provide high-fidelity results with far superior spatio-temporal coherency. On top of this, we are able to achieve SOTA results for video dehazing and relighting tasks and achieve SOTA among the test-time optimization techniques in UVOS tasks. We hope this clarification will alleviate the reviewer’s concern about the novelty issues.

Missing VOS baselines: In Table 2, we only compare with other test-time optimization baselines for a fair comparison as the setting of each methodology is exactly the same, i.e., no prior segmentation annotation is given to the model during training. We provide a comprehensive evaluation of the task UVOS task in Table 5 in the appendix because of the limited availability of space in the main paper. We will definitely add the remaining baselines[4,5,6] in there for a more comprehensive comparison. Additionally, if ICLR allows one extra page after acceptance, we would move those comparisons to the main paper.

Time taken: We refer the reviewer to our appendix section C.

General Guidance: We have explored three major downstream tasks like the VOS, dehazing, and relighting for videos. [1]* utilizes a test-time optimization technique for tasks like denoising, super-resolution, etc. Our framework can be naively extended to such tasks by utilizing similar loss functions; however, we leave this extension for future work.

[1]* `Video dynamics prior’ - Neurips 2023

With regards to the mentioned test-time methods[2,3], they are not at all designed for the VOS task but more for edit propagation tasks. These methods require per-frame masks obtained by image segmentation baselines. So, we do not see any overlap of these techniques with our framework apart from edit propagation tasks.

I was misunderstanding that your method also requires mask input for first frame. Thanks for the clarification.

We also would like to emphasize that maintaining spatio-temporal coherence is a big issue when it comes to tasks like dehazing and relighting, as can be seen from the qualitative results provided in the webpage(supplementary material). Our novelty also stems from the fact that our method can provide high-fidelity results with far superior spatio-temporal coherency.

See comments above - I found it really hard to say the proposed method has "far superior spatio-temporal coherency" from the video results; For relighting task, it seems all methods except SDSD has similar level of temporal coherency. Also, it is more important to understand why the proposed method has temporal coherency than just say "we have good results". For example, is the good temporal coherency comes from the $\alpha$-map extracted from optical-flow inputs; or it is a directly outcome of applying optical-flow as a warping constraint as a loss function? The latter (warping constraints) is a common approach in video tasks, though.

In Table 2, we only compare with other test-time optimization baselines for a fair comparison as the setting of each methodology is exactly the same, i.e., no prior segmentation annotation is given to the model during training.

Your setup is different from VOS as no initial mask is given. Thanks for clarification.

Time taken: We refer the reviewer to our appendix section C. I have read section C and still a little bit confused:

• "We achieve a processing rate of 60 frames/min for dehazing and relighting and 40 frames/min for VOS with one salient object"

Is this processing rate referring to optimization time or just inference time?

• "We use 100 epochs for a 60-frame sequence in VOS and 60 epochs for dehazing and relighting"

Let's make the answer simple - what exactly is the time for optimizing one video (60-frames, for example)?

We thank the reviewer XPSu for your thoughtful and detailed review of our paper on the Video Decomposition Prior (VDP) framework. We appreciate your recognition of our novel approach and its applications in video processing tasks. We have addressed each of your concerns below, aiming to clarify the aspects you highlighted. We believe these clarifications and additional information will solidify the strength of our paper and align it more closely with the acceptance criteria for ICLR.

1. Limitation of VDP not discussed: We acknowledge the importance of discussing the limitations of our methodology. To address this, we have added a comprehensive discussion on the limitations of VDP in Section H of our appendix (part of the supplementary material). Due to space constraints, this section was not included in the main paper, but we are prepared to move it to the main body if ICLR allows an additional page in the final submission. This inclusion will provide a more balanced view of our methodology and its applicability.

2. Flow Similarity Loss: We appreciate your query regarding the formulation of our Flow Similarity Loss. This loss is crucial for differentiating the motion of the background from the foreground. The process involves three steps:

• Step 1: We take the FlowRGB of the frame at time step $t$ (the process of obtaining FlowRGBs is detailed in Section K of the appendix included in supplementary material) and perform element-wise multiplication with both the foreground mask and the background mask, i.e.,

  • $FlowRGB_{Fg} = FlowRGB^t \odot M^t$
  • $FlowRGB_{Bg} = FlowRGB^t \odot (1-M)^t$

• Step 2: We use these obtained $FlowRGB_{Fg}$ and $FlowRGB_{Bg}$ and find their corresponding VGG embeddings. This is done by passing them as input to the VGG encoder (pretrained on ImageNet dataset and denoted as $\phi(.)$) and taking the output of the last layer of the VGG encoder as the embedding vector. Mathematically, we can write these embedding vectors for the foreground and background layer as,

  • $embed_{Fg} = \phi( FlowRGB_{Fg})$
  • $embed_{Bg} = \phi( FlowRGB_{Bg})$

• Step 3: In this final step, we calculate the loss value by taking a cosine value between these embeddings of the foreground and background layer.

  • $L_{FlowSim} = \frac{embed_{Fg} . embed_{Bg}}{|embed_{Fg}| |embed_{Bg}|}$

We minimize the cosine similarity between the VGG embeddings of the background layer and the foreground layer, which translates to the learning of a mask that separates the motion of the background and foreground. We hope this detailed explanation resolves any ambiguities about flow similarity loss.

3. Ablation Study Involving Reconstruction Layer Loss: We understand the concerns of the reviewer about the understanding the impact of reconstruction layer loss. We would refer the reviewer to section D.4 of the appendix(part of supplementary) where we perform extensive experiments on the DAVIS-16 dataset to study the impact of each loss term on the final segmentation results. We hope our clarifications have addressed the concerns of the reviewer.

We hope that our responses provide the necessary clarifications and insights into our methodology, reinforcing the potential impact and novelty of our work in the field of video processing. We are optimistic that these clarifications justify a rating upgrade from the current rating of "6".

We thank the reviewer for taking their time out of a busy schedule and evaluating our submission. We are glad that reviewer u6Cn found our paper well-motivated. Before we address the specific weaknesses and questions of the reviewer, we highly recommend reviewer u6Cn to please go through the webpage of video results, which was attached in the supplementary file.

Novelty issues:

We understand the concern raised by the reviewer over the novelty of the paper. Here is a more structured view of our model. The core of our framework for any task at hand has three components to it: prior knowledge(decomposition formulation), motion modeling (via $\alpha$-Net), and appearance modeling (via RGBnet). The framework’s way of modeling motion and appearance in a video is unique to our work. Specifically, to our knowledge, no prior test time optimization technique has utilized FlowRGBs to model the motion of the video.

Second, the utilization of prior knowledge(from gamma correction) to derive a novel decomposition formulation for the relighting task is a unique contribution of our work. Lastly, we are able to achieve state-of-the-art temporally coherent results utilizing our framework for the task of dehazing and relighting. We would urge the reviewer to please check out the difference in the qualitative video results of our methodology and baselines(We have provided these in an HTML webpage as a part of our supplementary material). To the reviewer’s point, our work derives the novelty from the above-mentioned points as opposed to the reconstruction or warping loss. We hope our clarification will alleviate the novelty concerns of the reviewer.

What is the consistency across tasks:

We thank the reviewer for pointing out the difficulty in understanding the consistency between the tasks. Hopefully, our clarification about it will alleviate this difficulty.

The core of our framework lies in its ability to adapt to different tasks while maintaining a consistent structure. Basically, each task consists of three integral components: prior knowledge (decomposition formulation), motion modeling (via $\alpha$-Net), and appearance modeling (via RGBnet). The consistent architecture of $\alpha$-Net and RGBnet establishes a unified way of extracting the motion and appearance features from a video across different tasks. These intermediate motion and appearance features are then utilized with task-specific adaptations that are reflected in the decomposition formulations. Hence, the unifying factor across different tasks is the consistency in the architecture used to extract motion and appearance cues from the videos across all tasks.

We would put more effort into revamping the introduction section of our manuscript to emphasize this point of consistency our framework's architecture brings across this variety of tasks.

Baseline comparison for UVOS Task:

We thank the reviewer for raising this point. In Table 2, we only compare with other test-time optimization baselines for a fair comparison as the setting of each methodology is exactly the same, i.e., no prior segmentation annotation is given to the model during training. We provide a comprehensive evaluation of the task UVOS task in Table 5 in the appendix because of the limited availability of space in the main paper. We would add the results to the main paper if ICLR allows one extra page in the final camera-ready paper.

Additional Experiments:

We provide more comparisons with the baseline suggested by the reviewers for the dehazing task. We utilize the MSBDN as an image dehazing baseline. Please find the findings below.

Additional Prior knowledge for the dehazing task:

We agree with the reviewer that additional knowledge about the scene can improve the efficacy of our methodology. However, it is very difficult to acquire hazy video and clean video pairs in a real-world setting. Hence, we formulated a test time optimization technique that does not rely on task-specific data for performing the dehazing.

We thank the reviewer 8QGm for your thorough review and constructive feedback on our submission, "Video Decomposition Prior (VDP)." We appreciate your recognition of VDP's innovative approach and its state-of-the-art performance in unsupervised video object segmentation, dehazing, and relighting. Your insights are invaluable, and we have addressed your concerns below, hoping to strengthen our paper and align it more closely with your expectations.

Addressing the Comprehensive Comparison Concern: You rightly pointed out the necessity for a comprehensive comparison with existing video editing techniques. To address this, we have included additional comparative analysis in Section G of our appendix(supplementary material). This section details a unique proxy task developed specifically to evaluate VDP against contemporary baselines, considering the absence of a standardized comparison framework in video editing. The results of this comparative analysis are presented in Table 6 of the appendix. Additionally, we have prepared a supplementary webpage featuring video results and qualitative comparisons with these baselines, which should provide a more tangible understanding of VDP's capabilities in relation to existing methods.

Refining the Title and Scope: In response to your suggestion, we agree that refining the title and scope of our paper could more accurately represent the specific editing tasks that VDP excels in. We propose the revised title, "Video Decomposition Prior and its Applications in Segmentation, Dehazing, and Relighting." This modification narrows down the scope and directly reflects the core strengths of our methodology. We are open to further suggestions from the reviewer regarding the title and scope adjustments.

We believe these updates and clarifications directly address your concerns and enhance the paper's relevance and contribution to the field. We hope these modifications and clarifications resonate with your assessment criteria and upgrade the paper's rating from a ‘6'.