Decoupled Finetuning for Domain Generalizable Semantic Segmentation

Thank you for the insightful and constructive comment! Below, we provide detailed responses to address your concerns.

Additional overhead in training time

To analyze additional computational and memory costs of DeFT, we measured the whole training time for 40K iterations and the peak GPU memory usage during the training. In both settings, the batch size is set to 4, and the cost is measured on GTAV with ResNet-50 backbone and only for the main training stage, except warm-up.

DeFT imposes additional space-time complexity only in training, and the overhead is not significant: 12% increase in training time and 22% increase in memory footprint, as shown in Table R1. This is because (1) the number of learnable parameters is exactly the same as that of joint finetuning and (2) the training images or labels are shared between the two pathways. Since GC is not updated by the standard gradient-based backpropagation, there is no need to compute or store gradients for GC. Moreover, since the two pathways share training data at every iteration, we do not need to load training data twice. These allow DeFT not to simply “double” the training time or memory usage. With such a small computation and memory overhead in training, it achieves significant performance improvement. Last but not least, DeFT does not have any additional overhead in the inference process.

Table R1. Comparison for the training time and peak GPU memory usage between joint finetuning and DeFT.

Sensitivity for EMA update ratio $\beta$

While the differences between the EMA update rates in Table 8 look tiny, even such small variations lead to significantly different temporal ensemble results. For example, the EMA update rates 0.999 and 0.9999 lead to the ensemble coefficients for the initial state 4.1683e-18 and 0.0183, respectively, after 40K iterations. This gap causes a substantial performance difference in DeFT where GCs should stay close to their initial states.

Target loss fluctuation

We measured the target loss fluctuations on Cityscapes during training on GTAV for DeFT. As images cannot be attached to the comment, we have added the result as Figure 9 in the appendix of the revised version of our paper, and would be grateful if you refer to it. As shown in Figure 9, DeFT significantly mitigates the target loss fluctuation, exhibiting consistently low target loss during training.

Thank you for the insightful and constructive comment! Below, we provide detailed responses to address your concerns.

Complexity of Implementation

First of all, the code-level implementation of DeFT is fairly simple, as described in Section A of the appendix. To analyze additional computational and memory costs of DeFT, we measured the whole training time for 40K iterations and the peak GPU memory usage during the training. In both settings, the batch size is set to 4, and the cost is measured on GTAV with ResNet-50 backbone and only for the main training stage, except warm-up.

DeFT imposes additional space-time complexity only in training, and the overhead is not significant: 12% increase in training time and 22% increase in memory footprint, as shown in Table R1. This is because (1) the number of learnable parameters is exactly the same as that of joint finetuning and (2) the training images or labels are shared between the two pathways. Since GC is not updated by the standard gradient-based backpropagation, there is no need to compute or store gradients for GC. Moreover, since the two pathways share training data at every iteration, we do not need to load training data twice. These allow DeFT not to simply “double” the training time or memory usage. With such a small computation and memory overhead in training, it achieves significant performance improvement. Last but not least, DeFT does not have any additional overhead in the inference process.

Table R1. Comparison for the training time and peak GPU memory usage between joint finetuning and DeFT.

Sensitivity analysis for the main parameters

In DeFT, the main parameters that have the biggest impact on performance are $\beta$ in Eq. (2) and $\alpha$ in Eq. (3).

1. Regarding $\beta$: The most important hyperparameter of DeFT is the EMA update rate $\beta$, which balances the pretrained knowledge and the task relevant knowledge of the GC. Results of the sensitivity analysis on this parameter were already reported in Table 8 of the paper.

2. Regarding $\alpha$: In the paper, we discuss the implicit regularization effect of DeFT on the distance from initialization, while also demonstrating that explicit regularization can be applied in conjunction. To this end, we employ a simple Euclidean distance-based regularization of Eq. (3), where the parameter $\alpha$ determines the contribution of the distance-based regularization. The impact of the explicit distance-based regularization has been already demonstrated in Table 3 of the paper. Please note that the explicit distance-based regularization is not the primary focus of our study; in the main tables of the paper comparing DeFT with prior work, the distance-based regularization has not been applied at all. So we did not provide an extensive ablation study on the impact of $\alpha$, but we are currently working on the analysis and will provide the results when they are ready.

The other hyperparameters such as the learning rate and weight decay were set to the values used in the previous work.

Experimental equipment

We conducted our experiments using four RTX 3090 GPUs.

Other update schemes for GC

Great suggestion. We conducted additional experiments by replacing the update scheme for GC in DeFT (i.e., EMA) with various weight ensemble methods.

(B) Simply averaging the latest UC weights with the initial weights, similar to WiSE-FT[1]

• $\theta^\text{GC}_t = 0.5 * \theta^\text{UC}_t + 0.5 * \theta_0$

(C) exponentially decreasing the later ensemble coefficient

• $\theta^\text{GC}_t = (\alpha * \theta^\text{GC}\_{t-1} + \theta^\text{UC}_t)/T, \quad T = \sum\_{i=0}^{t-1} \alpha^i, \quad \alpha > 1$

As shown in Table R2, the EMA biased towards initialization employed in DeFT clearly outperformed the other weight ensemble methods.

Table R2. Comparison between the EMA and other GC update schemes

References

[1] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

Thank you once again for your thoughtful review! Your feedback has been incredibly helpful in improving our work.

As the discussion period is nearing its end, we would be very grateful if you could take a moment to review the modifications and additional analysis we have made in response to your concerns. We hope these revisions address the points you raised.

Once again, we sincerely appreciate your valuable time and effort in reviewing our work.

Thank you for the insightful and constructive comment! Below, we provide detailed responses to address your concerns.

Theoretical foundation

Kumar et al. [1] showed that jointly finetuning a well-generalizable encoder with a randomly initialized decoder distorts the representation of the encoder, and proved this theoretically for the case of a two-layer linear neural network. To be specific, they demonstrated that such finetuning using only in-distribution (ID) data does not change the encoder's features for out-of-distribution (OOD) data although the encoder itself has been updated; this distortion deteriorates the encoder's generalization to OOD data. Further, such a distortion of one module also affects the other since loss gradients for updating the two are coupled; for example, the decoder's parameters are affected by the distortion of the encoder, which is referred to as "balancedness" in [2]. As a result, predictions of the entire model for OOD data could be corrupted. In other words, joint finetuning of the encoder and decoder on ID data (i.e., source domain data in domain generalization) distorts the generalizable knowledge of both modules, leading to poor generalization to OOD data (i.e., unseen target domain data in domain generalization).

On the other hand, consider a linear probing case where the encoder is fixed and no feature distortion occurs. Lemma A.14. in [1] shows that the upper bound of OOD error of the linear probing is proportional to the difference between the pretrained encoder and the "optimal" encoder, which shows the lowest errors for both ID and OOD data. That is, in the linear probing case, a decoder coupled with more generalizable encoder results in a more generalizable final model. Based on this, we first decouple the encoder and decoder to prevent each module from being distorted by their jointly finetuned counterparts, coupling them with counterparts which are not affected by distribution (i.e., domain) shift thus are well generalizable.

The optimization behavior from the perspective of individual network modules, such as encoders and decoders, as well as the interactions between these modules during the optimization process, seems to remain relatively underexplored. We expect that further foundational study in this direction might pave the way for more rigorous theoretical analysis of DeFT.

Versaltility of DeFT at a transformer backbone

We evaluated DeFT incorporating a transformer backbone, MiT-B5. As shown in Table R1, DeFT outperformed existing methods, such as DAFormer [3] and the combination of DAFormer and DGInStyle [4], using the same backbone. It is worth noting that DeFT achieved such outstanding performance without bells and whistles unlike DAFormer adapting its model architecture to domain generalization and DGInStyle relying heavily on extreme data augmentation; we strongly believe that employing such strategies will further improve the performance of DeFT. This result demonstrates that DeFT is a versatile training strategy applicable across various backbones.

Table R1. Comparison between DeFT and other methods incorporating MiT-B5 backbone. All models are trained on GTAV and evaluated on Cityscapes, BDD100K, and Mapillary.

References

[1] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations (ICLR), 2022.

[2] Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Proc. Neural Information Processing Systems (NeurIPS), 2018.

[3] Lukas Hoyer, Dengxin Dai, and Luc Van Gool. Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation. In Proc. IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2022.

[4] Yuru Jia, Lukas Hoyer, Shengyu Huang, Tianfu Wang, Luc Van Gool, Konrad Schindler, and Anton Obukhov. Dginstyle: Domain-generalizable semantic segmentation with image diffusion models and stylized semantic control. In Proc. European Conference on Computer Vision (ECCV), 2024.

Thanks for offering the latest manuscript. I have checked it. On several points of my concern, the article has certainly been improved. So I will raise my score to 6. Although the idea is straightforward, I believe this approach will become a new paradigm for training in the DG field in the future.

Justification for the evaluation settings

Sorry for the lack of clear justification for the evaluation protocol, which should have been covered in the paper at the time of submission. First of all, we chose the datasets and domain shifts to compare DeFT with most of existing domain generalization (DG) methods in the same settings since they have been widely used as the standard evaluation protocol for DG in the literature.

More importantly, the datasets and domain shifts enable us to simulate application scenarios that are intriguing and have great practical values. For example, {GTAV, SYNTHIA}→{Cityscapes, BDD100K, Mapillary} are synthetic-to-real generalization settings, and DG methods working in these settings may resolve the lack of labeled training data by exploiting synthetic data. Also, the Cityscapes→BDD100K setting can be used to evaluate a model’s robustness to distribution shifts in the wild by, for example, different geolocations and weather conditions.

Ablation experiments on the updated method

Thank you for the great suggestion! We conducted additional experiments on the two different update methods: using EMA-based updates (1) only for the encoder, and (2) only for the decoder. More specifically, we basically conducted joint finetuning of the encoder and decoder, yet replaced one module with its EMA version at each iteration, while keeping the other as is. These experiments were conducted on GTAV using ResNet-50 backbone while the EMA update rate $\beta$ was set to be 0.9999, the same as our DeFT. The result of these experiments is summarized in Table R1.

Table R1. Ablation experiments on the two updated methods

Please note that replacing a module with its EMA version at each iteration is equivalent to simply reducing the learning rate. Let $\theta_t$, $\theta'_t$ be a parameter and its EMA value at iteration $t$ respectively. Then, the $\theta_t$ and $\theta'_t$ are calculated as follows:

where $L$ is the loss calculated at iteration $t$ and $\lambda$ is a learning rate. However, $\theta'\_{t-1} = \theta\_{t-1}$, thus $\theta_t$ after the iteration $t$ is:

Hope we correctly understand your intention. Otherwise, could you please provide more details about the updated method in the comment? We will gladly conduct additional experiments to resolve your concern more precisely!

Thank you for the insightful and constructive comment! Below, we provide detailed responses to address your concerns.

More detailed theoretical explanation

Kumar et al. [1] showed that jointly finetuning a well-generalizable encoder with a randomly initialized decoder distorts the representation of the encoder, and proved this theoretically for the case of a two-layer linear neural network. To be specific, they demonstrated that such finetuning using only in-distribution (ID) data does not change the encoder's features for out-of-distribution (OOD) data although the encoder itself has been updated; this distortion deteriorates the encoder's generalization to OOD data. Further, such a distortion of one module also affects the other since loss gradients for updating the two are coupled; for example, the decoder's parameters are affected by the distortion of the encoder, which is referred to as "balancedness" in [2]. As a result, predictions of the entire model for OOD data could be corrupted. In other words, joint finetuning of the encoder and decoder on ID data (i.e., source domain data in domain generalization) distorts the generalizable knowledge of both modules, leading to poor generalization to OOD data (i.e., unseen target domain data in domain generalization).

On the other hand, consider a linear probing case where the encoder is fixed and no feature distortion occurs. Lemma A.14. in [1] shows that the upper bound of OOD error of the linear probing is proportional to the difference between the pretrained encoder and the "optimal" encoder, which shows the lowest errors for both ID and OOD data. That is, in the linear probing case, a decoder coupled with more generalizable encoder results in a more generalizable final model. Based on this, we first decouple the encoder and decoder to prevent each module from being distorted by their jointly finetuned counterparts, coupling them with counterparts which are not affected by distribution (i.e., domain) shift thus are well generalizable.

The optimization behavior from the perspective of individual network modules, such as encoders and decoders, as well as the interactions between these modules during the optimization process, seems to remain relatively underexplored. We expect that further foundational study in this direction might pave the way for more rigorous theoretical analysis of DeFT.

References

[1] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations (ICLR), 2022.

[2] Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Proc. Neural Information Processing Systems (NeurIPS), 2018.

Thank you once again for your thoughtful review! Your feedback has been incredibly helpful in improving our work.

As the discussion period is nearing its end, we would be very grateful if you could take a moment to review the modifications and additional analysis we have made in response to your concerns. We have also strengthened the theoretical foundation in Section B of the latest revision. We would greatly appreciate it if you could review it at your convenience. We hope these revisions address the points you raised.

Once again, we sincerely appreciate your valuable time and effort in reviewing our work.

Confusing writing, especially the terms GC and UC

We apologize for any confusion. We are continually striving to improve our writing. For better understanding, we have changed the terms Generalization Component (GC) and Usual Component (UC) to Retentive Component (RC) and Adaptive Component (AC), respectively. Please confirm these changes or let us know of any better alternatives, and we will revise the manuscript accordingly as soon as possible.

Theoretical foundation

Kumar et al. [1] showed that jointly finetuning a well-generalizable encoder with a randomly initialized decoder distorts the representation of the encoder, and proved this theoretically for the case of a two-layer linear neural network. To be specific, they demonstrated that such finetuning using only in-distribution (ID) data does not change the encoder's features for out-of-distribution (OOD) data although the encoder itself has been updated; this distortion deteriorates the encoder's generalization to OOD data. Further, such a distortion of one module also affects the other since loss gradients for updating the two are coupled; for example, the decoder's parameters are affected by the distortion of the encoder, which is referred to as "balancedness" in [2]. As a result, predictions of the entire model for OOD data could be corrupted. In other words, joint finetuning of the encoder and decoder on ID data (i.e., source domain data in domain generalization) distorts the generalizable knowledge of both modules, leading to poor generalization to OOD data (i.e., unseen target domain data in domain generalization).

On the other hand, consider the linear probing case where the encoder is fixed and no feature distortion occurs. Lemma A.14. in [1] shows that the upper bound of OOD error of the linear probing is proportional to the difference between the pretrained encoder and the "optimal" encoder, which shows the lowest errors for both ID and OOD data. That is, in the linear probing case, a decoder coupled with more generalizable encoder results in a more generalizable final model. Based on this, we first decouple the encoder and decoder to prevent each module from being distorted by their jointly finetuned counterparts, coupling them with counterparts which are not affected by distribution (i.e., domain) shift thus are well generalizable.

The optimization behavior from the perspective of individual network modules, such as encoders and decoders, as well as the interactions between these modules during the optimization process, seems to remain relatively underexplored. We expect that further foundational study in this direction might pave the way for more rigorous theoretical analysis of DeFT.

Additional computational and memory costs

To analyze additional computational and memory costs of DeFT, we measured the whole training time for 40K iterations and the peak GPU memory usage during the training. In both settings, the batch size is set to 4, and the cost is measured on GTAV with ResNet-50 backbone and only for the main training stage, except warm-up.

DeFT imposes additional space-time complexity only in training, and the overhead is not significant: 12% increase in training time and 22% increase in memory footprint, as shown in Table R2. This is because (1) the number of learnable parameters is exactly the same as that of joint finetuning and (2) the training images or labels are shared between the two pathways. Since GC is not updated by the standard gradient-based backpropagation, there is no need to compute or store gradients for GC. Moreover, since the two pathways share training data at every iteration, we do not need to load training data twice. These allow DeFT not to simply “double” the training time or memory usage. With such a small computation and memory overhead in training, it achieves significant performance improvement. Last but not least, DeFT does not impose any additional overhead in the inference process.

Table R2. Comparison for the training time and peak GPU memory usage between joint finetuning and DeFT.

References

[1] Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In International Conference on Learning Representations (ICLR), 2022.

[2] Simon S. Du, Wei Hu, and Jason D. Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Proc. Neural Information Processing Systems (NeurIPS), 2018.

Thank you for the insightful and constructive comment! Below, we provide detailed responses to address your concerns.

An ablation study to observe the effectiveness of decoupled finetuning alone for DeFT

As you suggested, we designed and conducted an experiment to investigate the effect of decoupled finetuning and that of weight ensemble separately. To be specific, we jointly finetuned the encoder and decoder, and considered their EMA versions as the final model for evaluation. The experiment was conducted using ResNet-50 backbone and the model was trained on GTAV. The EMA update rate $\beta$ was set to 0.9999, the same as DeFT.

As shown in Table R1, DeFT outperforms the aforementioned baseline (i.e., Joint + EMA). This result suggests that the proposed decoupled finetuning better preserves generalizable knowledge of the pretrained encoder and decoder than joint finetuning. Moreover, the result also shows that the temporal weight ensemble yields a more generalizable model than those trained only by the joint finetuning, which empirically justifies the use of the temporal weight ensemble as the update scheme for GC in DeFT.

Table R1. An ablation study to isolate and observe the effects of decoupled fine-tuning without the influence of temporal weight ensembling

Thank you once again for your thoughtful review! Your feedback has been incredibly helpful in improving our work.

As the discussion period is nearing its end, we would be very grateful if you could take a moment to review the modifications and additional analysis we have made in response to your concerns. We have also strengthened the theoretical foundation in Section B of the latest revision. We would greatly appreciate it if you could review it at your convenience. We hope these revisions address the points you raised.

Once again, we sincerely appreciate your valuable time and effort in reviewing our work.