EAGLE: Efficient Adaptive Geometry-based Learning in Cross-view Understanding

Dear Reviewer 29y1,

We would like to express our gratitude for your careful reading and valuable feedback. We are very happy you encourage that our paper addresses an important and practical problem, our proposed approach is well-tailored for the problem and appears to be solid and logical, and our approach also achieves solid quantitative performance. In addition, we would like to emphasize that other reviewers have also encouraged our quantitative results in our experiments and ablation studies are well-designed to illustrate the effectiveness of our proposed approach (Reviewer 8Etw, upJj, and JhPV). We appreciate your constructive comments and would like to address these points as follows.

[Q1] Lack of qualitative results

[A1] We have included the qualitative results of our ablation study in the pdf file of our rebuttal. In the model without prompting, Figure 1 illustrates the results of our cross-view adaptation compared to without adaptation. As shown in the results, our approach can effectively segment the objects in the drone view. We also compare with the prior ProDA [1] method in this figure. Our qualitative results remain better than the prior adaptation method. For the model with prompting, Figure 2 illustrates the effectiveness of our approach in 3 cases, without adaptation, with cross-view adaptation, and with view-condition prompting. As shown in the results, our cross-view adaptation can efficiently model the segmentation of the view. By using the view-condition prompting, our model can further improve the segmentation of persons and vehicles. The images are better viewed in color and 2x zoom. In addition to the quantitative results in Table 2, our qualitative results further confirm the effectiveness of our proposed approach. We will release our implementation for the reproducibility of both quantitative and qualitative results. We will release more qualitative results and comparisons in the final version of our paper.

[Q2] Using a VLM is the open-set setting

[A2] In Tables 5-6 in our paper, we report the performance of open-vocab segmentation (i.e. DenseCLIP, FreSeg). Specifically, The DenseCLIP model is developed based on the vision-language model (i.e., CLIP). These experimental results show the limited performance of these methods in the cross-view settings. The results also illustrate the effectiveness of our proposed approach in an open-vocab segmentation setting. In Table 6, we also present our experimental results in the open-set setting where classes of 'Tree' and 'Person' are considered as the unseen classes (which are not used during training). We would like to highlight that other reviewers encourage our experiments and ablation studies are well-designed for contributions (Reviewer 8Etw, upJj, and JhPV) and our experimental results are solid (Reviewers upJj, 8Etw, and JhPV).

References

[1] P Zhang, et al. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. CVPR, 2021.

Dear Reviewer 29y1,

As shown in the following table (the detailed comparison can be found in Table 4, page 8 in the paper), our EAGLE has outperformed CROVIA and ProDA. We believe the improvement compared to CROVIA is also illustrated in the visualization (as our comparison with ProDA) due to our reproduced results. In our revised paper, we will add more qualitative comparisons between our method and CROVIA and release our implementation for research reproducibility (both quantitative and qualitative results).

If you have any other concerns, please do not hesitate to raise your questions. We are happy to address your questions.

Thank you very much,

Authors

Dear Reviewer JhPV,

We greatly appreciate your insightful review and valuable feedback. We are very happy you encourage that our paper is well-written and features a good range of contributions, our problem is meaningful, and our idea is well-motivated. We appreciate your constructive comments and would like to address these points as follows.

[Q1] Layout of Paper

[A1] Thank you very much for your feedback. We will update the layout organization of our paper for better readability.

[Q2] Visual Separation in Tables

[A2] Thank you very much for your feedback. Per your suggestions, we will add more space to the tables for better readability.

[Q3] Free Up Space and Improve Presentation

[A3] Thank you very much for your feedback. We will update our paper according to your suggestions to improve our presentation and the readability.

Dear Reviewer 8Etw,

We are grateful for your careful reading and constructive feedback. We appreciate your highlighting that our proposed approach is efficient and well-designed, our problem is practical, and the method achieves solid performance. We appreciate your constructive comments and would like to address these points as follows.

[Q1] Additional Visualization Results

[A1] To further illustrate our qualitative results, we add more visualization results (Figure 1 and Figure 2) in the pdf file of our rebuttal. In addition, we will release more qualitative results in the supplementary of our paper. We will also release our implementation for the reproducibility of both quantitative and qualitative results.

[Q2] Generalizing the experiments to include more UAV datasets

[A2] We chose UAVID since these datasets have a great overlap with the source dataset (GTA, SYNTHIA, and BDD). We acknowledge the suggestion of the reviewer. Please refer to [KP3] for additional results of cross-view adaptation on UDD [1]. Due to the time limitation of the rebuttal period, we leave the experiments and investigation of other benchmarks of SynDrone [2] and ICG Drone [3] in our future work. It should be noted that we have also discussed the limitations of the dataset in the appendix.

[Q3] Performance of the building category in open-vocab segmentation on unseen classes

[A3] For clarification, in Table 5, all classes are used during training. In Table 6, we consider the classes of Tree and Person as the unseen classes. Therefore, the performance of building categories in our approach is valid and reasonable due to the effectiveness of our proposed method.

[Q4] Motivation behind the design of the adaptation loss

[A4] In our approach, we propose the adaptation loss by modeling the Cross-view Structural Change between source (car-view) and target (drone-view) domains. In particular, we first analyze the cross-view geometric correlation between two domains by analyzing the change of camera views and the equivalent transformation between image and segmentation output. Then, we propose to model the cross-view structural change via the geodesic flow. In particular, our loss measures the cross-view structural changes between the source and the target domains based on their manifold structures via the geodesic flow.

[Q5] The performance improvement in open-vocab segmentation is not very significant

[A5] We respectfully but strongly disagree with the reviewer on this point. As shown in Table 5, the performance of our open-cab experiment remains significant. For example, when we use our cross-view learning loss integrated with DenseCLIP (ViT) and FreSeg, the performance is up to 45.8% and 51.3 % on SYNTHIA $\to$ UAVID. Meanwhile, without our cross-view learning, the performance is only 21.8% and 23.4%. Our results still outperform prior adaptation methods (i.e., AdvEnt and SAC). Performance is even further improved when using our view-condition prompting.

[Q4] Misleading Notation

[A4] Thank you very much for your feedback. We will update our notation promptly.

References

[1] Y Chen, et al. Large-scale structure from motion with semantic constraints of aerial images. PRCV, 2018.

[2] G Rizzoli, et al. SynDrone – Multi-modal UAV Dataset for Urban Scenarios. ICCVW, 2023.

[3] ICG Drone Dataset. http://dronedataset.icg.tugraz.at/.

Dear Reviewer upJj,

We greatly appreciate your insightful review and valuable feedback. We are very happy you think our proposed approach is efficient and achieves significant experimental results. We appreciate your constructive comments and would like to address these points as follows.

[Q1] Comparison with other view transformation

[A1] The method in [1] requires depth for training which is unfaired compared to our approach. Please refer to [KP1] for our comparison with other view transformation methods.

[Q2] Results of Advanced Open-vocab Semantic Segmentation

[A2] The training code of clip-as-RNN [2] is not available. Please refer to [KP2] for our results using Cat-Seg [3]. Due to the time limitation of the rebuttal period, we leave the investigation of other open-vocab semantic segmentation [4, 5] in our future work.

[Q3] Difference of UDA settings between CROVIA and EAGLE

[A3] For the unsupervised cross-view adaptation settings, for fair comparison, we follow similar evaluation settings. However, the CROVIA [6] paper only focuses on the cross-view adaptation setting in semantic segmentation. Our work also considers the cross-view adaptation setting on open-vocab segmentation (Tables 5-7).

[Q4] Results of models with and without using cross-view setting

[A4] In Table 2, we have illustrated the performance with and without cross-view adaptation of open-vocab segmentation. In Table 5, we have also reported the performance of DenseCLIP and FreSeg without cross-view (the first row of each group in Table). Please refer to [KP2] for additional results of CatSeg.

[Q5] Using estimated camera matrices for the geometric correlation in Eqns (3) and (4)

[A5] Thank you very much for your suggestion. First, we clarify our cross-view modeling approach. From Eqn (3) to (4), it illustrates our approach to cross-view modeling by considering the transformation between two camera views. Then, Eqn (4) illustrates the necessary condition to explicitly model the cross-view geometric correlation under our analysis and assumption mentioned in L182-187. However, we acknowledge that using estimated camera matrices for the geometric correlation in Enq (3) and (4) could be a potential direction for further improvement. Due to the scope of our paper, we leave this investigation as our future work.

[Q6] Results of other domain adaptation methods on BDD $\to$ UAVID

[A6] In Table 7, we have reported the experimental results of cross-view adaptation setting on prior adaptation methods on BDD $\to$ UAVID, including BiMaL and DAFormer. Due to the time limitation of the rebuttal period, we leave the investigation of other DA methods on BDD $\to$ UAVID as our future work.

[Q7] Reason of DAFormer outperforming Mask2Former.

[A7] Our investigation reveals two reasons that could lead to the lower performance of Mask2Former compared to DAFormer. First, the network backbone of DAFormer is Transformer, while the backbone of Mask2Former in our experiments is ResNet 101. As shown in Table 1, if we use Swin as the backbone of Mask2Former, the results are improved compared to ResNet-101. Second, DAFormer adopts the convolution-based decoder. Meanwhile, Mask2Former adopted a Transformer for the decoder which could require more data for better generalization. Therefore, DAFormer in our experiments performs better than Mask2Former.

[Q8] Two subspaces of source and target domain and feature distributions before and after adaptation

[A8] In Remark 3, the Grassmannn manifold is presented and will be used to model the Cross-view Structural Change where the source domain is car-view and the target domain is drone-view. The two subspaces of source and target domain are obtained via the PCA algorithms. The details of subspaces are presented in our Implementation in the appendix (L638-L645). To illustrate the feature distributions, we use the features of the last layer before the classifier in the SYNTHIA $\to$ UAVID experiments. We compare our approach without and with cross-view adaptation. As shown in Figure 3 in the rebuttal pdf, our approach can help to improve the feature representations of classes, and the cluster of each class is more compact, especially in classes of car, tree, and person.

[Q9] Computational Complexity

[A9] In our proposed approach, the cross-view adaptation loss is only performed during the training. The computational time of our geodesic loss is approximately 0.0195 seconds per iteration. Meanwhile, in the practical deployment, the computational cost of our model relies on the segmentation network. The computational testing cost only relies on the segmentation network (e.g., DeepLab: ~776.2 GFLOPS and DAFormer: ~1447.6 GFLOPS).

[Q10] Target domain is limited in only one dataset

[A10] We have mentioned this limitation in our discussion. Please refer to [KP3] for results of an additional target domain.

References

[1] Pan, B., et al. Cross-view semantic segmentation for sensing surroundings. IEEE Robotics and Automation Letters, 2020.

[2] Sun, S. et al. Clip as rnn: Segment countless visual concepts without training endeavor. CVPR, 2024.

[3] Cho, S., et al. Cat-seg: Cost aggregation for open-vocabulary semantic segmentation. CVPR 2024.

[4] Zou, X., et al. Generalized decoding for pixel, image, and language. CVPR, 2023.

[5] Wysoczańska, M., et al. Clip-dinoiser: Teaching clip a few dino tricks. arXiv, 2023.

[6] T.D. Truong, et al. CROVIA: Seeing Drone Scenes from Car Perspective via Cross-View Adaptation, arXiv 2024.