Hyperbolic Active Learning for Semantic Segmentation under Domain Shift

Dear Reviewer 3ouZ, Your commitment to reviewing our paper is greatly appreciated. Kindly find our responses to your questions below:

• (Weakness 1) This question plays a crucial role in understanding the novel interpretation of the hyperbolic radius and the proposed acquisition strategy, so we have addressed it in our message to all reviewers. Please see the answer on Correlation Analysis. Additionally, following the reviewer's advice, we have computed the correlation between the class accuracy and the percentage of total pixels, which yields 0.539. This correlation is moderate, but lower than that between the hyperbolic radius Vs.\ the class accuracy and total number of pixels, $-0.605$ and $-0.899$ respectively. In our answer on Correlation Analysis, as further support to the use of the hyperbolic radius as a data acquisition strategy, please also see the larger correlations in the case of Cityscapes $\rightarrow$ ACDC, -0.759 and -0.868 respectively.

• (Weakness 2) Thank you for your comment. As highlighted in the message to all reviewers, we value your feedback and have diligently attended to the noted writing issues, conducting additional proofreading to enhance the overall quality of the paper. Check out the Paper's writing and detailing for the answer.

• (Question 1) We agree on the intriguing potential of extending HALO for Active Learning to other tasks. In fact, we are actively engaged in research on active learning for image classification and object detection. However, we must acknowledge that this presents several challenges and complexities, which need careful considerations:

  • Divergent Task Requirements: Our current work draws from the insights of Hyperbolic Image Segmentation (Atigh et al. 2022). For the object detection task an entirely distinct pipeline is necessary and no prior work is available. On the other hand, image classification with HNNs has proven effective primarily on datasets with inherent hierarchies and self-supervised learning approaches (Ermolov et al. 2022, HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization Kim et al. 2022). Notably, the challenge arises from the fact that the significance of the hyperbolic radius was demonstrated at the pixel level for semantic segmentation tasks, whereas it becomes meaningful at the patch or full-image level primarily in the context of hierarchical relationships among classes. Our objective is to generalize the utility of hyperbolic learning without imposing hierarchical labels or relying on inherently hierarchical datasets with hyperbolic loss functions.
  • Dataset Imbalance: An additional consideration lies in the observed high correlation between the hyperbolic radius and class difficulty in cases of highly imbalanced datasets. However, this correlation may not hold for datasets such as CIFAR-10/-100 or ImageNet, where class distributions tend to be more balanced.

  Given these substantial disparities between the HNNs literature and the tasks at hand, extending HALO for active learning to these domains, while intriguing, would indeed represent a significant endeavor that surpasses the intended scope of this paper.

Dear Reviewer kZFp, Your dedicated time and effort in reviewing our paper are deeply appreciated. Enclosed are our responses addressing your questions and concerns:

• (Weakness 1) We are sorry for the confusion caused by insufficient pipeline description and we appreciate the opportunity to clarify the details. Following [1,2], HALO comprises a Euclidean Segmenter (DeepLab-v2 or DeepLab-v3plus), a hyperbolic projection layer (expmap), and a hyperbolic multinomial logistic regression (HyperMLR) layer. The Euclidean Segmenter produces a d-dimensional embedding (d=64 for HALO) for each pixel of the image, which is projected onto the Poincaré ball via expmap, thus yielding the hyperbolic embedding of each pixel. The parameters to estimate the hyperbolic embedding for each pixel are end-to-end learned, based a pixel classification loss (cf. the HyperMLR layer.) To answer your question: embeddings are end-to-end learned in a hyperbolic manifold for each pixel. We have updated these details in the HALO pipeline in Sec. 5.1. Fig. 2 is generated by calculating the Class Average Radius, which involves computing the mean hyperbolic radius for all pixel embeddings within each class in the target dataset. Additionally, the Class Accuracy in Fig. 2 (left) represents the average classification accuracy for each class in the target dataset. Finally, the % of Total Pixels in Fig. 2 (right) corresponds to the percentage of acquired pixel annotations at the end of the active learning process for each class.

• (Weakness 2) To the best of our knowledge, RIPU [6] is the state-of-the-art on ADA for SS on the two benchmarks of GTAV $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Cityscapes. The two benchmarks were originally adopted by MADA [7] (Ning et al. ICCV'21), which proposes the task of ADA for SS for the first time. Our claim is supported by surpassing RIPU on both benchmarks and on a third one, which we additionally introduce, Cityscape $\rightarrow$ ACDC.

• (Weakness 3) Let us note that Fully Hyperbolic Neural Networks (Fully-HNN) have not gained widespread adoption nor have they demonstrated to be on par with alternative approaches. We are aware of the following Fully-HNN: Poincaré ResNet [4], Hyperbolic RNN [5], and Hyperbolic MLP [5]. They do not yield the state-of-the-art in the benchmark tasks for which they were proposed. Also, none of them has been benchmarked on semantic segmentation, so far. This is likely due to issues on convergence and computational load, which is beyond the purpose of this paper to address. Our methodology is rooted in previous frameworks, such as [1,2,3], where the hyperbolic neural network is conventionally divided into an Euclidean model and subsequent hyperbolic projection and classification layers. In this process, a feature vector $v$ is indeed extracted from the Euclidean space before being projected into the Poincaré ball. This approach aligns with the current hyperbolic state-of-the-art method for semantic segmentation [1].

• (Weakness 4 and Question 1) Your inquiry is much appreciated as it regards the specific reason why HALO achieves state-of-the-art performance, in our view. Specifically, HALO's strategy involves the selection of boundary pixels for "easier" classes such as car or road, while also incorporating internal pixels for more complex classes such as person, rider, and bicycle. HALO showcases the importance of boundary pixels while acknowledging the significance of inner parts of objects in comprehending the scene. Please refer to Section A.6 in the appendix and Fig. 7 for more insights. To validate this, we conduct an oracle study involving the exclusive selection of boundary pixels during active learning stages. However, the resulting performance of 69.8% mIoU was inferior to both HALO and RIPU. Please take a look at the "Baseline selection comparison" response in the message to all reviewers for more details about HALO's pixel selection.

• (Question 2) Thank you for this question, as it allows us to remark the interesting insight of the proposed analysis. Usually, in hyperbolic space, when the (1 - radius) is interpreted as the uncertainty, a low radius indicates high uncertainty. However, in our case, classes with lower radii, such as road, sky, building, are actually easier to classify by the model (higher accuracy and mIoU), leading to a mismatch with the canonical interpretation of hyperbolic uncertainty. Effectively, our work had originated by the wish to leverage the relation between the hyperbolic radius and uncertainty, but it transformed into a novel geometric interpretation of the hyperbolic radius when we realized a novel interpretation was needed. The nuanced interpretation of the hyperbolic radius is extensively discussed in Sec. 4, where we also provide a comparison with existing viewpoints (see Sec. 4.2). While a theoretical proof remains still an open research question, our empirical evidence currently supports this novel interpretation.

• (Question 3) Thank you for your question. Our approach is not dependent on manually defined class hierarchies. In fact, this study originates from retraining the architecture of [1] and finding that the interpretation of the hyperbolic radius changes when not enforcing hierarchies.

7. (Question 4) This query is crucial to understanding the empirical motivation of the novel interpretation of the hyperbolic radius and the proposed HALO data acquisition strategy, so we have addressed it in the message to all reviewers, under "Correlation Analysis". Please refer to it.

[1] Atigh et al., "Hyperbolic Image Segmentation," CVPR 2022.

[2] Ermolov et al., "Hyperbolic Vision Transformers: Combining Improvements in Metric Learning," CVPR 2022.

[3] Franco et al., "Hyperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representations," ICLR 2023.

[4] van Spengler et al., "Poincaré ResNet," ICCV 2023.

[5] Ganea et al., "Hyperbolic Neural Networks," NIPS 2018.

[6] Xie et al., "Towards Fewer Annotations: Active Learning via Region Impurity and Prediction Uncertainty for Domain Adaptive Semantic Segmentation," CVPR 2022.

[7] Ning et al., "Multi-Anchor Active Domain Adaptation for Semantic Segmentation", ICCV 2021

4. We appreciate your comment, as it prompted us to enhance the comprehensiveness of our paper. In Appendix A.8, we have included additional comparisons with RIPU, that comprehend a qualitative assessment of the selection process and present quantitative results from the oracle experiment to elucidate the limitations of RIPU's boundary selection. Please refer to the "Baseline Selection Comparison" section in the message to all reviewers for the complete answer.

5. Thank you for raising this point, as it prompted us to conduct a more comprehensive evaluation of these approaches. In response, we test Guo et al., 2022's Feature Clipping method in our framework for comparison with our HFR. As shown in the table below, while Feature Clipping works and produces better results than the baseline RIPU, it still falls short of our HFR method (-1.2%). Guo et al. utilize Feature Clipping to prevent vanishing gradients during backpropagation. Despite its simplicity, this technique restricts the model's representational capacity by clipping features, resulting in inferior performance compared to HFR. In the allotted rebuttal time, we have not tested the curriculum learning of [Franco et al., 2023] and the initialization approach of [van Spengler et al., 2023], because their adaptation to the ADA task is not straightforward. The curriculum learning in Franco et al. is specifically tailored for metric learning scenarios involving a hyperbolic loss, enabling training in hyperbolic space by utilizing cosine distance for improved initialization, gradually transitioning to the Poincaré loss. Our method does not involve comparing embeddings and leveraging the Poincaré loss. Similarly, the initialization approach in van Spengler et al. is designed explicitly for fully hyperbolic ResNets, particularly hyperbolic convolutions. As we do not employ hyperbolic convolutional layers, their initialization approach is not immediately suitable for our model.

   Method	mIoU
   Hyperbolic Feature Reweighting (ours)	74.5
   Feature Clipping (Guo et al., 2022)	73.3
   Initialization (van Spengler et al., 2023)	not compatible
   Curriculum Learning (Franco et al., 2023)	not compatible

6. We genuinely appreciate your question and the opportunity it provides to clarify certain aspects of our proposed method. In our paper, we utilize the entropy of the softmax probability array as a measure of uncertainty (ref. Eq. 7), drawing from the existing literature on ADA [Gal et al., 2017; Wang & Shang, 2014; Wang et al., 2016; Sinha et al., 2019; Xie et al., 2022b; Prabhu et al., 2021; Xie et al., 2022a], which currently yields state-of-the-art results. We acknowledge that relying solely on the pure softmax value may not accurately estimate model uncertainty, as you point out. However, please consider that our contribution regards the introduction of the hyperbolic radius to complement uncertainty as the data acquisition strategy. Maintaining the entropic softmax formulation of earlier literature and esp. of RIPU enables a fair comparison. We agree that uncertainty deserves more consideration in future research.

Dear Reviewer UCCT, We sincerely appreciate the time and effort you dedicated to reviewing our paper. Below, you'll find our responses to your questions and concerns:

1. Thanks for the feedback. We've addressed the writing issues and we've conducted thorough proofreading of the paper. This has involved rectifying the indexes of Figures 2, 3, and 4, ensuring consistent labeling with "Fig." We've also refined certain sentences to enhance the overall clarity of the paper.

2. We sincerely appreciate your insightful question and the opportunity to further elucidate the correlation analysis presented in Section 4.1. Please consider the answer "Correlation Analysis", reported in the message to all reviewers, where we present a more detailed analysis of the correlations. In this answer, we discuss some additional insights on the Region- Vs. Pixel-based criteria, following your suggestion. While the region impurity score of RIPU requires pixel regions, as the impurity is based on region statistics, the hyperbolic radius employed in HALO can be computed on both pixel and region bases. In response to your suggestion, we train HALO with the region-based approach. As we can observe in the following table, the region-based approach leads to a small difference of -0.4% on the GTAV $\rightarrow$ Cityscapes benchmark with 5% acquired labels, but still manages to achieve a significative improvement over the baseline (RIPU).

   Method	Pixel-based	Region-based
   RIPU	-	71.2
   HALO	74.5	74.1

3. Your query holds significant importance and offers valuable insight into comparing parameters between Euclidean and Hyperbolic Neural Networks. To delve deeper into this comparison, we conduct an assessment of the parameter count between our method HALO and the prior state-of-the-art, RIPU [Xie et al.]. Both employ the DeepLab-v3+ architecture but with some distinctions:

   1. RIPU operates with a pixel embedding dimension of 512, resulting in a parameter count of 60.1M.
   2. In contrast, HALO operates with a reduced pixel embedding dimension of 64, which the adoption of a hyperbolic learning enables. Moreover, the HyperMLR requires fewer parameters than the Euclidean Linear layer used for classification due to the reduced embedding dimension. This results in a slightly lower total parameter count than RIPU's (10k fewer params). Additionally, HALO introduces the HFR module, consisting of two linear layers separated by a BatchNorm layer and a ReLU. Thanks to the lower embedding dimensions, the input and output sizes of the HFR module are only 64-dimensional, adding less than 10k additional parameters. This roughly matches the number of parameters removed from the segmenter. These modifications result in the parameter count being nearly identical between the two methods (60.1M), aligning with other studies leveraging the DeepLab-v3+ architecture. Here's a table for a better visual comparison between the models:

   Method	Segmenter	Dim.	HFR (params)	Total Params
   RIPU	DeepLab-v3+	512	Not used	60.1M
   HALO	Hyper-DeepLab-v3+	64	10k	60.1M