MaskTwins: Dual-form Complementary Masking for Domain-Adaptive Image Segmentation

Thank you for your valuable time and comments. The main concerns are addressed below.

W1: Ablation study of the used components

Regarding your concerns, we fully ablate the used components on MitoEM-H → MitoEM-R in the following tables.

---

Table R1. Ablation study of each loss component on MitoEM-H → MitoEM-R. The mean and standard deviation are computed over 3 random seeds. $L_{sup}$ = supervised loss, $L_{cl}$ = consistency loss, $L_{cm}$ = complementary masked loss, $L_{rm}$ = randomly masked loss, with a mask ratio of 0.5.

---

As shown in Table R1, adding the consistency loss $L_{cl}$ improves the performance over the supervised loss. Further, we separately incorporate the randomly masked loss $L_{rm}$ and the complementary masked loss $L_{cm}$. The results indicate that both losses contribute to performance improvements, with our proposed complementary masking strategy being more effective than the random masking strategy. The results with $L_{rm}$ and $L_{cm}$ have been visually shown in Figure 4 in the main paper.

---

Table R2. Ablation study of EMA and AdaIN on MitoEM-H → MitoEM-R. The mean and standard deviation are computed over 3 random seeds.

---

In our work, we adopt the EMA teacher to keep consistent with previous methods, such as CAMix [1], MIC [2], DAFormer [3], etc. The results show that both EMA and AdaIN contribute to performance improvements, with EMA having a more significant impact. To address any potential misunderstandings caused by the use of these components, we will include these experiments and necessary descriptions in the revised paper.

Dear Authors,

Thanks for your efforts to take into account my comments.

Unfortunately, it seems that the proposed method exhibits very marginal improvements based on Tables R1 and R2. Meanwhile, the overall framework is derived from many well-constructed modules, showing incremental contributions. Therefore, I maintain my score 5 and raise the confidence to 4.

Thanks for your efforts again.

Q2.1: The folds/runs of the evaluation results

Our results are averaged over 3 random seeds, following [5, 6]. We do not use cross validation since the target domain (test set) is fixed in the setting of UDA tasks.

Q2.2: The stability of training

We provide the mean and standard deviation of the experiments as below. For natural image segmentation and mitochondria segmentation, we are not able to include comparison results with other methods because their original papers did not report precise standard deviations, and some methods have not been open-sourced. For synapse detection, we provide the results of other methods since these are reproduced by us. The results indicate that our method demonstrates high stability across different runs, with low variance in performance metrics. This demonstrates the robustness and reliability of our training process.

---

Table R3. Natural Image Segmentation results on SYNTHIA → Cityscapses. For more details, see Table 1.

---

Table R4. Mitochondria Segmentation results. For more details, see Table 2.

---

Table R5. Synapse Detection results.

---

Reference

[1] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In NIPS, pages 137–144, 2006.

[2] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1-2):151–175, 2010.

[3] Zhang, Yuchen, et al. "Bridging theory and algorithm for domain adaptation." International conference on machine learning. PMLR, 2019.

[4] Zhou, Qianyu, et al. "Context-aware mixup for domain adaptive semantic segmentation." IEEE Transactions on Circuits and Systems for Video Technology 33.2 (2022): 804-817.

[5] Hoyer, Lukas, et al. "MIC: Masked image consistency for context-enhanced domain adaptation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[6] Hoyer, Lukas, Dengxin Dai, and Luc Van Gool. "Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Thank you for your valuable time and comments. The main concerns are addressed below.

W1: Connection to UDA in the theoretical analysis

As suggested, we will briefly discuss how our complementary masking strategy relates to the theory of domain adaptation in the revised paper to further enrich our theoretical background.

The theoretical works [1, 2, 3] provide fundamental insights into UDA, especially concerning domain discrepancy and theoretical bounds. Specifically, they study margin bounds for classification tasks at the distribution level, while we focus on segmentation tasks and the theory of Masked Image Modeling and compressed sensing at the image level. We have analyzed the information preservation, generalization bounds and feature consistency to demonstrate the effectiveness of complementary masking. The paper [3] which is grounded in the theory of domain adaptation such as [1] and [2] also discusses generalization bounds based on empirical Rademacher complexity. We preliminary observe that there exists deeper connections between these works and ours. Hopefully, we will make further theoretical analysis in the next version.

As for the "naive masking strategy", it serves as a control group to ensure a fair comparison with our complementary mask. We have included comparisons to the random masking strategy (widely adopted in mask-related works such as MAE) in our ablation study, demonstrating that our complementary masking provides significant benefits over random masking. We validate in the mask type section of our ablation study the significant advantages of complementary masking compared to random masking.

W2 & Q1.1: Comparison with random baseline and ablation study of loss components

First, we have compared the complementary masking with the random baseline from the theoretical analysis by conducting an ablation study on mask types in the main paper, and we provide the results again below. Next, as suggested, we fully ablate different components of losses on MitoEM-H → MitoEM-R in the following table.

---

Table R1. Ablation study of each loss component on MitoEM-H → MitoEM-R. The mean and standard deviation are computed over 3 random seeds. $L_{sup}$ denotes the supervised loss, $L_{cl}$ is the consistency loss, $L_{cm}$ is the complementary masked loss, and $L_{rm}$ is the randomly masked loss, with a mask ratio of 0.5.

---

As shown in Table R1, adding the consistency loss $L_{cl}$ improves the performance over the supervised loss. Further, we separately incorporate the randomly masked loss $L_{rm}$ and the complementary masked loss $L_{cm}$. The results indicate that both losses contribute to performance improvements, with our proposed complementary masking strategy being more effective than the random masking strategy. The results with $L_{rm}$ and $L_{cm}$ have been visually shown in Figure 4 in the main paper.

Q1.2: The benefit of EMA teacher-student framework and the effect of not including AdaIN

The EMA teacher realizes a temporal ensemble of previous student models, which increases the robustness and temporal stability of pseudo-labels. It is a common strategy used in semi-supervised learning and UDA. In our work, we adopt the EMA teacher to keep consistent with previous methods, such as CAMix [4], MIC [5], DAFormer [6], etc.

We further ablate the effect of the EMA teacher and AdaIN in the following table. The results show that both EMA and AdaIN contribute to performance improvements, with EMA having a more significant impact.

---

Table R2. Ablation study of EMA and AdaIN on MitoEM-H → MitoEM-R. The mean and standard deviation are computed over 3 random seeds.

---

W3: Ablation study of the used components

As suggested, we fully ablate the used components on MitoEM-H → MitoEM-R fully in the following tables.

---

Table R1. Ablation study of each loss component on MitoEM-H → MitoEM-R. The mean and standard deviation are computed over 3 random seeds. $L_{sup}$ = supervised loss, $L_{cl}$ = consistency loss, $L_{cm}$ = complementary masked loss, $L_{rm}$ = randomly masked loss, with a mask ratio of 0.5.

---

As shown in Table R1, adding the consistency loss $L_{cl}$ improves the performance over the supervised loss. Further, we separately incorporate the randomly masked loss $L_{rm}$ and the complementary masked loss $L_{cm}$. The results indicate that both losses contribute to performance improvements, with our proposed complementary masking strategy being more effective than the random masking strategy. The results with $L_{rm}$ and $L_{cm}$ have been visually shown in Figure 4 in the main paper.

---

Table R2. Ablation study of EMA and AdaIN on MitoEM-H → MitoEM-R. The mean and standard deviation are computed over 3 random seeds.

---

The results show that both EMA and AdaIN contribute to performance improvements, with EMA having a more significant impact.

Reference

[1] Shin, Ukcheol, et al. "Complementary random masking for rgb-thermal semantic segmentation." 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.

[2] Zhou, Qianyu, et al. "Context-aware mixup for domain adaptive semantic segmentation." IEEE Transactions on Circuits and Systems for Video Technology 33.2 (2022): 804-817.

[3] Hoyer, Lukas, et al. "MIC: Masked image consistency for context-enhanced domain adaptation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[4] Hoyer, Lukas, Dengxin Dai, and Luc Van Gool. "Daformer: Improving network architectures and training strategies for domain-adaptive semantic segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Thank you for your valuable time and comments. The main concerns are addressed below.

W1: This paper is similar to [1]. Therefore, it seems that the complementary masking techniques are not novel.

(1) Both papers propose complementary masking techniques to promote robustness.

Thank you for mentioning [1], which is an inspiring work and we will cite it in the revised paper. However, our work is inherently different from [1] in terms of the masking definition and the application scope. More importantly, we have provided theoretical analysis of our complementary masking strategy.

1. The masking definition is different. [1] performs complementary masking strategy on two modalities: RGB and thermal images. In contrast, we focus on a single modality to ensure that both the masks and the generated masked images are complementary. Therefore, the definition of complementary masking in our work is more flexible.
2. The application scope is different. [1] focuses on RGB-Thermal semantic segmentation, tackling the over-reliance on one modality in multimodal inputs. The masking technique in [1] is tightly coupled with RGB-thermal pairs and relies on these specific multimodal datasets. In contrast, our work addresses the unsupervised domain adaptation (UDA) problem, which involves adapting models across domains with unlabeled target-domain data. Our complementary masking strategy is applied to single-modality inputs, which makes it more generalizable and widely applicable across tasks and datasets.
3. We have strictly proved that complementary masking can effectively extract domain-invariant features. In our work, complementary masking is framed as a dual-form masking strategy grounded in sparse signal reconstruction theory. This theoretical foundation is a core contribution of our work, as it rigorously proves the advantages of complementary masking in preserving information, reducing variance, and improving feature consistency.

(2) Both methods use consistency mechanisms to enforce feature learning.

The "consistency mechanism" is a generic concept and does not indicate significant similarity. Consistency mechanisms are widely used in semi-supervised learning and UDA tasks as a fundamental strategy to promote robust feature learning. While both works adopt consistency mechanisms, our work leverages dual-form complementary masking consistency as a novel way to enforce feature learning across complementary views, specifically tailored for unsupervised domain adaptation. This is a natural progression in UDA tasks and does not imply direct overlap with [1].

In conclusion, while [1] provides a practical solution for RGB-Thermal segmentation, our work builds a rigorous theoretical framework for complementary masking and applies it to the broader, more challenging problem of domain adaptation. We appreciate the opportunity to clarify these distinctions in the revised paper.

W2: The dependence on pseudo-labels

We agree that pseudo-labeling introduces inherent noise due to the distribution gap between the source and target domains. However, this is a well-established challenge in UDA tasks. Addressing this, we emphasize the following points:

1. Pseudo-label quality is not the core focus of our work. To avoid distracting from our main contributions—complementary masking and consistency learning—we adopt the EMA teacher model for pseudo-label generation, following common practices in UDA methods such as CAMix [2], MIC [3], and DAFormer [4]. This ensures fair and consistent comparison with existing state-of-the-art methods.
2. Our complementary masked loss reduces dependence on pseudo-labels. Unlike traditional UDA methods that rely heavily on pseudo-label quality for learning, our dual-form complementary masking strategy introduces an additional component of robustness. Specifically, the complementary masked loss encourages the model to make consistent predictions across masked views without the supervision of pseudo-label, which mitigates the impact of noise during training. This is evident in Table R1, where the complementary masked loss improves segmentation performance even in the challenging UDA scenario.
3. EMA pseudo-labeling is widely adopted and effective for UDA. While pseudo-labeling introduces noise, it has been shown to be an effective strategy when combined with consistency regularization. Our work builds on this foundation and demonstrates that complementary masking further enhances robustness, as reflected in our experimental results.