Dynamic Shadow Unveils Invisible Semantics for Video Outpainting

Q1: The relationship between shadows and video outpainting.

R1: Even when an instance (e.g., an object) goes outside the frame, its shadow often stays in the video and with useful cues such as motion and action. Traditional video outpainting methods often overlook shadows in videos or treat them as noise, which is unreasonable. In contrast, our method explicitly models the dynamic shadows and incorporates this information as a spatiotemporal cue to enhance the spatiotemporal consistency of the generated video.

---

Q2: The ground truth or the detected shadow.

R2: We did not use ground truth shadow annotations, and adopted DAVIS and YouTube-VOS datasets even do not include the ground truth of shadow. Besides, the quality of detected shadows can impact performance, as shown in R-Tab 5.

R-Tab 5: Sensitivity analysis of our method to different confidence thresholds used in the shadow detector.

We conducted experiments using multiple threshold settings of the video instance shadow detector. Our method remains robust across different thresholds, which shows our method can handle the impact of initial object-shadow detection from the video instance shadow detector.

---

Q3: The better visualizations for the visual result.

R3: Thank you for your valuable suggestion to improve the visualizations in Fig 3 and 4. We will incorporate the input images in the final version along with the current captions that explicitly indicate the masked regions to provide better visual reference and enhance clarity.

---

Q4: The hidden semantics of dynamic shadow in the video.

R4: Instances such as objects and shadows are not merely visual features but also carry rich semantic information. Specifically, shadows are 2D projections of objects and often implicitly convey rich semantic information about objects. (1) In static images, shadows can unveil the category and spatial location of objects.
(2) In videos, they further unveil the actions and motion of the objects over time.
Therefore, modeling shadows is not just a visual consideration but also a way to capture the hidden semantics of objects in the scenes.

Dear Reviewer ak5T,

We are grateful that our response has addressed your concerns and deeply appreciate your positive score for our work.

We also thank you for your constructive feedback and will thoughtfully incorporate your comments, together with the additional analyses, into the revised manuscript.

Warm regards,

The Authors of Submission 19821

Q1: Difference between MOTIA and our method.

R1: The differences between MOTIA and our method are significant. Specifically, MOTIA relies on test-time adaptation by training a LoRA module for each test sample, which means they need to spend considerable time training on every video during the inference stage. In contrast, our method does not require retraining on each video, making a comparison unfair. Nevertheless, the performance comparison on DAVIS remains comparable.

---

Q2: The performance in scenarios with a moving camera.

R2: As can be seen from Fig 1, 3 and 4, these several samples are from moving camera scenarios. And the issue of dramatic changes in shadow caused by the relative movement of the light source can be addressed by our method. Because our module observes the shadow-object pairs over time to infer the changing relative position of the light source. In addition, the spatiotemporal fusion strategy can implicitly incorporate the understanding of the changing relative position of the light source into the refinement of each shadow-object pair.

Note that in both the DAVIS and YouTube-VOS datasets, most videos have a moving camera, and only a few are shot from a fixed position. To further address your concern about handling dramatic changes in shadows, we conducted an additional experiment using videos from the YouTube-VOS dataset, as can be seen from R-Tab 6.

R-Tab 6: Quantitative comparison on video in different camera motion settings from the Youtube-VOS dataset.

The relative position of the light source and the scale of the shadow distortion are hard to quantify directly since there are no ground-truth shadow annotations in our adopted datasets. We therefore measure the camera motions instead, in which large camera motions between two frames are more likely with largely distorted shadows. The inter-frame camera motion is first measured by Structure-from-Motion (SfM). Then, for a single video, if its average per-frame translation is less than 5 Euclidean distance and average per-frame rotation angle is less than 5 degrees, we label it as a slow-motion video, otherwise a fast-motion one. As can be seen from R-Tab 2, although the fast-motion videos should be with largely distorted shadows, our method performs robustly on them.

---

Q3: The effectiveness of the shadow prior in our pipeline.

R3: Our method shows the effectiveness of the shadow prior, as can be seen from Fig 3 and Tab 1 through comparisons with M3DDM. As MOTIA performs test-time adaptation by training a LoRA for each test sample, which incurs large inference-time computation. Our method achieves better performance than M3DDM, which also does not require any test-time retraining. Furthermore, when our proposed components are integrated, our model outperforms MOTIA, demonstrating that the shadow prior in our method is effective.

---

Q4: The novelty of video-aware shadow alignment discriminator.

R4: It not only enhances the spatiotemporal consistency of dynamic shadow-object pairs but also enhances scene-level generation, even generating a reasonable light source, as can be seen from Fig 1(c). Modules like the instance adapter and flow adapter focus on capturing instance-level motion and scene-level coherent motion individually, to enhance video consistency. However, video generation goes beyond just handling moving instances; it also requires generating a realistic and semantic-related background, including accurate lighting and scene semantics. During the decoding process from latent space to video space, our discriminator contributes to generating more realistic shadows and background illumination compared to the pre-trained decoder of Stable Diffusion.

---

Q5: The details of the user study.

R5: We conducted a user study to compare our method with M3DDM and MOTIA on the YouTube-VOS dataset with a horizontal mask ratio of 0.66. A total of 40 volunteers participated in the study, and each volunteer evaluated 20 randomly selected result sets. Participants were asked to choose the best result under each of the two criteria: visual quality (including clarity, color fidelity, and texture detail) and temporal consistency (including motion smoothness, object continuity, and temporal coherence).

---

Q6: The method for resolving the overlapping shadows cast by multiple objects.

R6: We introduce an inter-pair fusion module and a relaxed shadow mask supervision strategy to address the issue of overlapping shadows cast by multiple objects. Specifically, the inter-pair fusion module enables spatial information sharing across different object-shadow pairs. Additionally, a relaxed shadow mask supervision strategy does not penalize predictions containing connected regions of other shadows, encouraging the model to learn to complete the shadow regions of each object in complex multi-object scenarios.

---

Q7: The performance comparison of adapter-based training versus full U-Net fine-tuning in diffusion models.

R7: Full U-Net fine-tuning outperforms adapter-based training, as can be seen from R-Tab 7.

R-Tab 7: Comparison of different fine-tuning strategies on the shrunk Stable Diffusion on Youtube-VOS.

However, due to computational resource constraints, we are unable to fine-tune the full U-Net of Stable Diffusion. Instead, we choose to freeze the backbone and train only the adapters, which still achieves competitive performance.

---

Q8: The sensitivity of optical-flow completion when the camera pans quickly and shadow smear.

R8: Our module is not sensitive to such cases as fast camera panning and shadow smearing, as can be seen from R-Tab 8.

R-Tab 8: The sensitivity of optical-flow completion under different camera motions.

Note that, unlike video deblurring, the task of video outpainting involves clear input frames. Therefore, shadow smear caused by fast motion is not an issue in our task setting. Even under fast camera motion, our optical-flow completion still performs well due to the object-shadow mask providing instance-level spatial guidance.

Dear Reviewer wJGo,

We are grateful that our response has addressed your concerns and deeply appreciate your positive score for our work.

We also thank you for your constructive feedback and will thoughtfully incorporate your comments, together with the additional analyses, into the revised manuscript.

Warm regards,

The Authors of Submission 19821

Q1: The generalizability of the method to videos with or without shadows.

R1: Our method performs well on videos both with and without shadows, as can be seen from R-Tab 3.

R-Tab 3: Quantitative comparison on videos with and without shadows from the Youtube-VOS dataset.

Note that, for a single video, if the shadow area is less than 10% of the single frame size and lasts for less than 20% of the video duration, we label it as a video without shadows, otherwise a video with shadows.

When shadows are absent (i.e., the shadow mask is all zeros), our method still performs conventional outpainting, which indicates that our model is more general, rather than specific to scenarios with shadows.

---

Q2: The computational overhead of the proposed method relative to baselines in training and inference.

R2: Our method offers a good trade-off between speed and quality during inference, as can be seen from R-Tab 4.

R-Tab 4: Comparison of model size, trainable parameters, and inference time across different methods on Youtube-VOS.

Because the training code for M3DDM has not been publicly released, we have re-implemented it for testing. And MOTIA requires training a LoRA for each test-time sample, which incurs significant per-instance computational cost during inference preparation. As mentioned in the supplementary material, we randomly selected 10 videos for outpainting at a resolution of 256×256. On a single RTX A6000 GPU, M3DDM required approximately 48 seconds per video, MOTIA took about 8 minutes, and our method completed the task in approximately 55 seconds per video.

---

Q3: The way to derive the global feature $h^t$ of a single frame.

R3: The global feature $h^t$ is obtained by concatenating individual instance features $h^{t,n}$, as can be seen from Eq 16. In addition, the number of instance pairs is capped at a fixed maximum of 10 to ensure consistent input dimensions, as detailed in the supplementary material.

---

Q4: The design of the learnable vector $z^0$ in the inter-frame fusion module.

R4: This learnable vector $z^0$ is shared by each pair to extract their global embedding $h^0$ individually. Such an aggregation structure is similar to that used in Vision Transformer (ViT)[1], which has been demonstrated to be effective. This learned unique latent code $z^{0,n}$ is assigned to each shadow-object pair to derive the global embedding $h^{0,n}$. Then, to avoid confusion across different pairs, the global embedding $h^0$ of each pair is not involved in the inter-pair fusion process, as can be seen from Eq 7.

[1] "An image is worth 16x16 words: Transformers for image recognition at scale." In the International Conference on Learning Representations (ICLR) 2021.

Dear Reviewer uqHS,

We hope that our response has addressed the concerns raised in your feedback, and we sincerely appreciate your decision to update the score accordingly.

If you have any additional concerns or would like to continue the discussion, we would be happy to engage further.

Thank you once again for your thoughtful and constructive feedback.

Best regards,

The Authors of Submission 19821.

Dear Reviewer uqHS,

We are grateful that our response has addressed your concerns and deeply appreciate your decision to increase the score for our work.

We also thank you for your constructive feedback and will thoughtfully incorporate your comments, together with the additional analyses, into the revised manuscript.

Warm regards, The Authors of Submission 19821

Q1: The generalizability of the method to real-world videos.

R1: Our method performs well on real-world videos because shadows commonly exist in natural scenes. As can be seen in the supplementary material, videos from real-world datasets like DAVIS and YouTube-VOS have an average of 0.9 and 1.2 shadow-object pairs per video, with each pair lasting around 38.5 frames in DAVIS and 24.2 frames in YouTube-VOS. Moreover, for 76% of videos with shadow-object pairs, the object becomes invisible while its shadow is still visible.

It can be seen from R-Tab 1 and R-Tab 2 that our method performs robustly on the cases with less prominent shadows (R-Tab 1) and heavily distorted shadows (R-Tab 2). This is because our proposed spatio-temporal fusion strategy alleviates the effect of low-quality shadow detection in single frames by considering the entire video.

R-Tab 1: Sensitivity analysis of our method to different confidence thresholds used in the shadow detector.

In R-Tab 1, we change the confidence threshold in the shadow detector, where a low threshold means detections with low confidence will be preserved and used in the following outpainting process. Usually, the less prominent shadows have low confidence, and thus can be filtered out by setting a high confidence threshold. Even though some of the less prominent shadows cannot be filtered out by setting a high confidence threshold, our method still performs robustly on them. As can be seen from R-Tab 1, our method is not sensitive to the quality of the shadow detection, and performs well even when setting the confidence threshold to be 0.2, in which situation most of the low-quality shadow detections are preserved while the outpainting quality decreases slightly.

R-Tab 2: Quantitative comparison on video in different camera motion settings from the Youtube-VOS dataset.

The scale of the shadow distortion is hard to quantify directly since there are no ground-truth shadow annotations in our adopted datasets. We therefore measure the camera motions instead, in which large camera motions between two frames are more likely with largely distorted shadows. The inter-frame camera motion is first measured by Structure-from-Motion (SfM). Then, for a single video, if its average per-frame translation is less than 5 Euclidean distance and average per-frame rotation angle is less than 5 degrees, we label it as a slow-motion video, otherwise a fast-motion one. As can be seen from R-Tab 2, although the fast-motion videos should be with largely distorted shadows, our method performs robustly on them.

---

Q2: The impact of the initial object-shadow detection.

R2: As can be seen from R-Tab 1, our method can handle the impact of initial object-shadow detection even when the confidence threshold in detection is set to be low, where lots of detected shadows with low confidence are used for outpainting. This is because our spatio-temporal fusion module is specifically designed to refine the initial detection results, thus enhancing the system’s robustness to errors such as imprecise masks or overlapping shadows.

---

Q3: The training strategy of the modular components.

R3: The shadow-aware instance prediction module and the instance-aware optical flow completion module are pre-trained separately. Then, the entire framework is trained together in an end-to-end manner.

Dear Reviewer G7tk,

We are grateful that our response has addressed your concerns and deeply appreciate your positive score for our work.

We also thank you for your constructive feedback and will thoughtfully incorporate your comments, together with the additional analyses, into the revised manuscript.

Warm regards,

The Authors of Submission 19821