End-to-End Video Semantic Segmentation in Adverse Weather using Fusion Blocks and Temporal-Spatial Teacher-Student Learning

We thank the reviewer for recognizing that our task is important, the idea is novel and contributes to the community, and with a good performance.

Weakness-1: The white area is not involved in computing the spatial loss, we will adjust the arrow to make it correctly pointing to the right-bottom part.

Weakness-2: Thank you for pointing out the issue. Your understanding is correct, and we will revise it accordingly.

Weakness-3: The weight smooth coefficient is 0.9995. Regarding the pseudo-label selection for the teacher-student losses, we use the ratio of pixels exceeding a threshold τ (0.968) of the maximum softmax probability, as suggested in [7].

Weakness-4: Unlike the Cityscapes dataset, which contains images captured under clear conditions using an automotive-grade 22 cm baseline stereo camera and produces high-quality images, the MVSS dataset is more suited to real-world scenarios. The MVSS images are captured from different cameras under various conditions and are of relatively lower quality. The Cityscapes dataset features images with a resolution of 2048 x 1024, whereas the MVSS dataset images have a resolution smaller than 630 x 460. Due to this lower quality, the qualitative results from MVSS inputs do not appear as good as those from Cityscapes.

Weakness-5: Thank you for the suggestion, we will include Accel in the related work and emphasis the importance of inference speed.

Weakness-6: For the “foggy” and glare effects, we first randomly select the affected area in the current frame. To create the “foggy” area, we adjust the gamma and chromaticity. The glare effect is produced by using a Gaussian kernel at the selected area, with the center having the highest values that gradually decrease with distance from the center. Once the augmentations in the current frame are completed, we add the augmentations to the adjacent frames. However, the affected area will randomly move around, and the intensity of the augmentations will also vary, mimicking real-world scenarios.

Weakness-7: Thank you for the suggestion and we will follow accordingly.

We thank Reviewer 1Ffg for recognizing that our work can benefit the community, novel, effective, and with a significant performance gain.

Q1: The MVSS dataset consists of a total of 52,735 RGB images, with 3,545 of these images annotated. The dataset includes a variety of adverse conditions such as overexposure, nighttime, rain, fog, and snow.

Q2: In the teacher-student pipeline, the weights of both teachers are updated using the following equation: $W_t(i+1) = 0.9995W_t(i) + (1 - 0.9995)W_s(i)$, where $W_t(i)$ represents the weight of the teacher models at iteration $i$, and $W_s(i)$ represents the weight of the student model at iteration $i$. For the weighing factors of the teacher-student losses, we use the ratio of pixels exceeding a threshold $\tau \ (0.968)$ of the maximum softmax probability, as suggested in [7]. For all other configurations, we use the same settings as those in the existing methods described in [5, 27] to ensure a fair comparison.

Q3: We appreciate the suggestion. We will expand the related work and discuss all the suggested papers.

Q4: We have included a more recent method for more qualitative comparison in the attached rebuttal.pdf.

Q5: Thank you for pointing out the presentation issues in the paper, we will revise all of them accordingly.

Q6: We appreciate the suggestion. We will further improve the presentation of the images.

We thank Reviewer L8p7 for recognizing the importance and performance of our work. Here is our response to the feedback:

Weakness-1:

Thank you for the feedback. Our model has an inference time of 0.11 seconds, which is faster than the state-of-the-art baselines: DA-VSN's 0.35 seconds and TPS's 0.17 seconds. In terms of parameters, our model’s size is 173MB, whereas DA-VSN is 619MB (FlowNet part) + 169MB (segmentation part), and TPS is 619MB (FlowNet part) + 169MB (segmentation part). Note that, although TPS is a faster and more accurate version of DA-VSN, both share the same number of parameters.

Weakness-2 and Weakness-4:

Thank you for pointing this out. In the main paper, we selected TPS for visual comparison because it has the second-best quantitative performance. In addition to this, we recently found a CVPR’24 paper (published after the NeurIPS’24 submission deadline), but its code has not yet been released. We implemented it ourselves following the motion training provided by the authors. We will include this CVPR’24 paper and the related discussion stated in this rebuttal in our main paper. Additionally, we will add a more thorough survey of this year’s publications to our main paper.

Here is the qualitative comparison. For IoU, higher values are better.

For Table 1, Viper to MVSS domain adaptation,

For Table 2, Synthia to MVSS domain adaptation,

We have also included the qualitative comparison in the attached rebuttal PDF.

Pan et al., “MoDA: Leveraging Motion Priors from Videos for Advancing Unsupervised Domain Adaptation in Semantic Segmentation”, CVPR’24

Weakness-3:

We have made every effort to address the reviewer’s concerns regarding the lack of in-depth discussion and theoretical justification, which we will elaborate below. However, with all due respect, we are unsure which specific parts of our method require deeper analysis and further theoretical justification. More detailed feedback on this would be much appreciated.

Existing optical flow methods often fail under adverse weather conditions. To address this issue, we use a temporal teacher to guide the student network in learning from adjacent frames, instead of relying on optical flow. Here is our basic idea. We input the current frame into the temporal teacher, which generates predictions used as pseudo-labels. Concurrently, we mask out part of the current frame and feed it into the student network. We enforce a consistency loss between the student’s predictions and the pseudo-labels from the temporal teacher. This approach encourages the student network to reconstruct the masked-out information by leveraging temporal data from adjacent frames, so that it can align its predictions with the pseudo-labels.

To ensure that the student network does not overlook important spatial information while focusing on temporal data, we incorporate a spatial teacher. This integration helps the student model learn and utilize relevant spatial information as well.

As objects move, their features can appear in different locations across frames, making feature alignment challenging as we don’t have optical flow information. To address this, we use a fusion block with deformable convolutional layers to align features from different frames, followed by standard convolutional layers for merging. This block is trained end-to-end with the temporal teacher, enabling the network to effectively integrate temporal information from various frames and improve the model’s semantic segmentation performance.

Weaknesses-5: Thank you, we will follow the suggestion.

We sincerely thank the reviewer for the further feedback.

Point 1: Comparisons with SFC and MoDA

Parameters

SFC: 406 MB

MoDA: 185MB (Motion part) + 169MB (Segmentation part)

Ours: 173MB

Inference Speed

SFC: 0.38s+0.35s (SFC requires two stages, the first for generating robust optical flows, and the second for segmentation)

MoDA: 0.19s+0.17s (Similar to SFC, MoDA also requires two stages)

Ours: 0.11s

All the models we experimented with were run on a single RTX 3090.

Point 2: Utilizing deformable convolutions

We agree that deformable convolutions to align features is common. We do not actually claim it is our novelty. Our novelty is integrating our fusion block that use deformable convolution into our temporal-spatial teacher-student framework so that optical-flow-free video-based semantic segmentation can be achieved. The deformable convolutions serve as a tool to bridge information across different frames. We will add this to our paper for clarity.

Point 3: Describe the pipelines with mathematical formulas

We agree with the reviewer and will follow the suggection. Essentially, we plan to include the following discussion and would greatly appreciate any further feedback.

Let the input image at frame $i$ be denoted as $X_i$, with the student encoder as $S$ and the teacher encoder as $T$. We define the student fusion block as $F_S$ and the teacher fusion block as $F_T$. For the temporal pipeline, we enforce the following consistency:

$F_S(S(A_{TWD}(X_{i-1})), S(\text{Crop}(A_{TWD}(X_{i}))))=F_T(T(X_{i-1}), T(X_i))$

Here, $A_{TWD}$ represent the temporal weather degradations, and Crop indicates that the model is provided with only a cropped segment of the current frame. By enforcing this consistency, we encourage the student model to align with the teacher’s performance. As a result, the student model learns to be robust against weather degradation while effectively utilizing information from $X_{i-1}$ to compensate for missing details in the cropped current frame.

For the spatial pipeline, we enforce the following:

$S(\text{Crop}(A_{TWD}(X_{i})))=T(\tilde{X}_i)$

where $\tilde{X}_i$ represents the same cropped image segment at a higher resolution. By enforcing this consistency, we ensure that the student model remains robust to weather degradation while preserving spatial precision.

Once again, we appreciate your valuable suggestions. We will incorporate the proposed mathematical equations and other feedback into our paper to enhance clarity for our readers.

We thank Reviewer G5E9 for recognizing the importance of our task, our good results, and the clarity of our paper.

Weakness-1 and Weakness-2

Thank you for your valuable feedback. We will address both Weakness-1 and Weakness-2 together, as they are closely related. Additionally, the novelty of our fusion block should be considered an integral part of our end-to-end framework for temporal modeling, rather than a standalone module.

The innovation of our Temporal-Spatial Teacher-Student learning approach lies in providing a solution for temporal modeling without the use of optical flow, particularly in scenarios where ground truths for the target domain are unavailable. In our case, since a significant part of the current frame is completely erased, there are no clues left for the student model to learn from. Consequently, the student model must extract temporal information from adjacent frames to fill in the erased parts and produce a consistent prediction with the temporal teacher’s pseudo labels. Unlike existing methods that use optical flow to warp temporal information, our approach enables the student model to learn to gather temporal information through this teacher-student learning mechanism. During the learning process, the spatial teacher ensures the spatial precision of the model and may also enhance it, preventing the network from focusing too much on temporal information and neglecting the spatial information in the current frame.

Since we intend to avoid using optical flows under adverse weather conditions, we need to design an alternative solution to merge temporal information for the student network. Therefore, our fusion block is designed to "offset" and "fuse" information from different frames. Objects in consecutive frames can move to different locations due to their motion. For instance, a car in the bottom-left corner of frame t might move to the top-right corner in frame t+1. Directly concatenating features from such frames can cause misalignment due to large displacements. Existing methods use optical flow to "offset" objects between frames before concatenating features. Instead of using optical flow, we integrate deformable convolutional layers with standard convolutional layers. The deformable convolutional layers learn an "offset" to relocate features, followed by standard convolutional layers to "fuse" the relocated features, as illustrated in Figure 2(c). This combination allows our network to gather temporal information without any pretrained optical flows and enables end-to-end training with the Temporal-Spatial Teacher-Student learning approach under adverse weather conditions.

To the best of our knowledge, integrating temporal and spatial modeling using two teachers and one student to achieve an optical-flow-free model is novel. Additionally, the fusion block and its integration into our temporal teacher-student framework have not been explored before. To avoid any misunderstanding, we will clarify this in our paper

Weakness-3:

Regarding the fairness of our comparison, the backbone for all the models is the same, using AdvEnt, as stated in the experiment section. Apart from the components discussed in our paper, we did not add any other elements to our experiments.

Under clear weather conditions, accurate optical flow can indeed provide strong prior information for generating better predictions. However, under adverse weather conditions, existing optical flow methods can be erroneous, leading to degraded performance. Existing methods use optical flows generated from pretrained FlowNet 2.0 [20], which is trained on clear synthetic datasets and thus become erroneous when applied to adverse weather scenes. Although we could augment clear data with synthetic adverse weather conditions, it is well known that there are significant gaps between synthetic and real weather conditions. Figure 3 in the main paper shows that under clear daytime conditions (left columns), the optical flows are accurate and useful for semantic segmentation. However, under adverse weather conditions (right columns), the optical flow predictions are erroneous. Consequently, using generated optical flows can introduce incorrect information and decrease the performance of existing models.

Weakness-4:

Following the suggestion, we will expand the related work section to include a video semantic segmentation subsection, covering [a-c] and other relevant works.

Our use of temporal information differs from that in methods [a-c], which are conventional video semantic segmentation tasks trained with ground-truth labels from datasets like VSPW. In those methods, models learn to capture temporal information from consecutive frames using supervision from these labels. In contrast, our paper focuses on self-supervised domain adaptive video semantic segmentation, which does not rely on ground-truth labels.

Without ground-truth supervision, domain adaptive methods cannot directly learn temporal information. As discussed in our paper, existing methods [4, 5, 18, 23, 27] use optical flow to warp predictions from consecutive frames to generate pseudo labels, thereby “forcing” the model to learn temporal information. However, generated optical flow can be erroneous under adverse weather conditions, as evidenced in Figure 3. Therefore, we develop an optical-flow-free approach to compel the model to learn temporal information.