Automated Label Unification for Multi-Dataset Semantic Segmentation with GNNs

Thank you for your thorough review and insightful comments regarding our manuscript. Below are our responses to your questions and concerns.

W1: Focus on benchmark performance vs. taxonomy quality

We acknowledge the importance of reasoning within a unified label space. Our focus on multi-dataset training aims to automatically integrate labels across datasets. Evaluation on these datasets is a straightforward approach. To assess the quality of our unified label space, we conducted an indirect evaluation using WD2, demonstrating its effectiveness in unseen scenarios.

W2: Impact of training exclusively on driving datasets

We recognize the potential concern regarding dataset bias. It's worth noting that other methods on the WD2 leaderboard also use datasets beyond RVC. In the revised version, we will include a model trained on the full RVC collection for a more comprehensive comparison.

W3: Difficulty isolating individual training design choices

We apologize for the lack of clarity in Table 2. Due to long training times and limited access to open-source methods, comprehensive comparative experiments are challenging. Our primary comparison highlights the significant improvement our model offers over the single dataset and multi-SegHead baselines.

W4: Lack of discussion on adjacency matrix initialization

We regret this omission and have conducted ongoing experiments to address it. Results indicate that our proposed initialization method (Algorithm 2) provides a slight improvement over randomized initialization on a 3ds setting, suggesting a relatively minor impact on overall performance.

---

Q1: Size of the created taxonomy in different tables and performance after initializing the adjacency matrix.

Thank you for your suggestion. We agree that including the initialization of the adjacency matrix in Table 5 would enhance reader understanding. However, due to time constraints, we cannot currently provide results for all seven datasets. These findings will be included in the revised version of the paper. Comparative experiments in a smaller setting were conducted, as addressed in our response to Weakness 4. Below is a summary of label space sizes for the methods, which we will incorporate into the revised version.

Q2: Comparison of created taxonomy with the manual taxonomy.

To compare our constructed taxonomy with the manual taxonomy provided by [5], we adopted [5] as the ground truth standard, assessing whether categories from all datasets were appropriately linked. Categories were categorized into Merged, Single, and Split classes, each evaluated based on Correct, Partially Correct, and Wrong Classifications. We introduced Wrong but reasonable to accommodate justified connections that were nonetheless classified as wrong. For instance, while the Cityscapes dataset marked the Traffic sign (back) as ignored, [5] connected it based on expert knowledge.

The summary of our 7ds learned taxonomy is as follows:

An initial taxonomy calculated using Algorithm 2 yields:

Notably, our constructed unified label space contains only 217 categories, representing a 6% reduction from the 231 categories formed by Algorithm 2’s initial adjacency matrix, while achieving nearly consistent quality in taxonomy recovery.

Q3: Mapping of unseen datasets to the unified taxonomy.

As indicated in Lines 212-214 of our paper, the mapping process is conducted automatically. We identify the optimal mapping by comparing the unified label categories predicted by our model on the training set of the unseen datasets against the ground truth labels.

Q4: Advantages of using the formulation in Eq. 6 compared to NLL+ [4].

We regret that because [4] is not open-sourced, our reproduced version of NLL+ does not converge under our training framework, prompting us to utilize Eq. 6 for training instead.

Q5: Differences between trained taxonomy and initialized taxonomy.

For the differences in the label space, please refer to Q2 and the global rebuttal. The initial unified label space had 231 categories, but some became obsolete during training, resulting in 217 categories in the final 7ds-trained model. Our paper (L487-480) explains the methods to eliminate inactive connections. Specifically, during the final training phase, we evaluate the model in the training dataset to remove inactive connections and unlinked nodes.

Q6: Explanation of the zero-shot model on WildDash 2.

The zero-shot model on WildDash 2, referenced in Table 4, is the model trained using our 7 datasets (i.e., the "ours" model in Table 2). We evaluated this model against the WildDash 2 dataset without any additional fine-tuning.

Q7: Time required to initialize the adjacency matrix and comparison with [52].

Using four 80G A100 GPUs, initializing the adjacency matrix takes approximately two days to train the Multi-SegHead model, followed by half a day of cross-evaluation across multiple datasets. Obtaining the initial adjacency matrix using Algorithm 2 takes around one hour. In comparison to [52], we modified the cost calculation to utilize the IoU metric. This approach is only applicable to category merging and shares similarities with the partial merge method in [5].

We hope this clarifies your questions and appreciate your feedback.

We are pleased to inform you that we have obtained the complete experimental results regarding the initialization, answering W4 & Q1. Our experiments were conducted on three datasets (Cityscapes, Sunrgbd and CamVid) using two 32GB V100 GPUs. After completing 50,000 multi-SegHead training iterations, we performed 50,000 segmentation network training iterations for each group using the same multi-head model parameters. For the experiments involving GNN, an additional 30,000 GNN training iterations were included (without updating the segmentation network parameters). The results indicate that our GNN training shows a performance improvement compared to the results obtained using Algorithm 2 for initializing the adjacency matrix without GNN training. Algorithm 2 provides a good starting point for GNN training and contributes to the overall performance enhancement.

Dear Reviewer,

Thank you for your valuable feedback. We agree that evaluating a universal taxonomy instead of merely benchmark performance of downstream tasks is a crucial aspect of this research, particularly in terms of relation discovery quality. However, we currently face a challenge due to the lack of a dedicated benchmark specifically designed for assessing taxonomy quality. While the manually curated taxonomy provided by [5] serves as a helpful reference, it also incorporates expert knowledge that might not be well aligned with the visual features present in the dataset images. Furthermore, there is currently an absence of well-established metrics specifically for this type of evaluation. Given these current limitations, this work has primarily focused on evaluating model performance across different datasets. Moving forward, we are committed to exploring ways to better evaluate relation discovery quality and to address the concerns you have raised. We appreciate your insights and believe they are instrumental in guiding our future research endeavors.

Regarding the WD2 performance, our zero-shot model achieved SOTA results compared to other zero-shot models on the leaderboard, which we believe highlights the contributions of our approach. It’s important to note that the Vistas dataset lacks annotations for the pickup, van, and autorickshaw categories present in WD2. Additionally, we did not utilize the relabeled data provided by WD2 for these categories in the MVD, Cityscapes, and IDD datasets during training. Therefore, given the dataset bias, it is expected that the zero-shot model would perform lower on WD2 compared to models specifically trained on the WD2 dataset.

Thanks!

Thank you very much for taking the time to review our work. Below, we summarize each of your questions and provide detailed responses.

Q1 & Q2: Concerns about the lack of detailed explanation and the omission of the latest methods

A: We respectfully disagree with your assessment. Our approach is fundamentally different from existing methods and two methods you mention. While existing methods require manual re-labeling (e.g., MSeg [23]), expert knowledge (e.g., NLL+ [4]), or are limited to only two datasets (e.g., Auto Univ. [6]), our method automatically constructs a unified label space across multiple datasets without human intervention or iterative processes. Furthermore, we are the first to leverage Graph Neural Networks for this task.

The papers you mentioned do not construct a unified label space. Instead, they utilize different segmentation heads (with different weights) for evaluation across various datasets. For instance, [1] employs a text encoder to encode label categories into corresponding embedding spaces for each dataset, predicting within dataset-specific embedding spaces. This method also struggles with handling categories that share the same name but have different annotation standards across datasets (e.g., IDD "curb" vs. MPL "curb"). This challenge is precisely why we introduced $d_i$ in Equation (1), allowing nodes with the same name from different datasets to obtain distinct node features, thereby differentiating these nodes.

On the other hand, [2] uses dataset-specific batch normalization and heads, which can be heavily influenced by dataset biases. In real-world inference scenarios, the outputs from different segmentation heads may conflict, complicating practical application. Our method, however, consistently predicts across different datasets using a unified label space (with the same weights), employing different boolean label mapping matrices solely for performance evaluation. In practical inference, this label mapping is unnecessary. Therefore, methods [1-2] require prior knowledge of the target label space for predictions, which does not align with our task setup and is not suitable for comparison.

Q3: Suggestion to explore scalability on transformer-based networks.

A: Thank you for your valuable suggestion. We acknowledge the lack of experiments exploring the effectiveness of our method across different models. Training on seven datasets requires approximately one week on four 80G A100 GPUs, which limits our ability to provide results using transformer-based networks at this time. However, we plan to investigate the scalability of our method with transformer-based architectures in future work.

Q4: Concerns about the significance of improvements from label descriptions and resource consumption.

A: We appreciate your feedback regarding the use of GPT for generating text descriptions. It is important to clarify that the process of generating these descriptions using the text encoder is not conducted in real-time; it involves a one-time inference step. The time cost for this step is less than one minute, and the space requirement for preserving text features is only 3.5MB, which is negligible compared to the overall training costs. Furthermore, after training, the GNN component is discarded, meaning it does not introduce any additional overhead during the inference phase.

We hope these clarifications address your concerns effectively.

Dear Reviewer,

Thank you for your thoughtful feedback and for acknowledging our efforts in the rebuttal. To the best of our knowledge, the SOTA method we refer to is the one presented in IJCV 2024 [5], where our approach demonstrated significant performance improvements in both the 7ds and WD2 ( 51.3 vs. 56.4 on 7ds and 46.9 vs. 50.2 on WD2). Could you please clarify if the SOTA methods you are referring to are those mentioned in your initial review? If so, we have already emphasized that these methods are not directly comparable within our dataset-agnostic setting, as they only provide dataset-aware prediction and are unable to provide predictions in a unified label space. If you are referring to other methods, we would appreciate it if you could provide some examples for our reference.

We would also like to emphasize that we did not intend to avoid comparisons. Rather, it was challenging to produce the comparison results within the limited time frame. We plan to include additional experimental data in the revised paper, such as results on WD2 using the full RVC collection for training and on 7ds with the initialized adjacency matrix without GNN training. As for your suggestion to include Transformer-based methods, we greatly appreciate this valuable input. We will certainly consider exploring and discussing these approaches in a future version of the paper.

Thanks! We look forward to your any further feedback.

Thank you for your valuable feedback and for highlighting the areas that require clarification in our paper. We appreciate your insights, which allow us to improve our manuscript. Below, We summarize each of your questions and provide detailed responses.

Q1: Clarification on how graph connectivity is learned and visual similarity is utilized.

A: Thank you for your question. The node features in our approach only contain textual characteristics. The visual information, is provided by the segmentation model during training. Specifically, the graph neural network outputs embeddings and adjacency matrix that are utilized in the segmentation network's inference process. For the image training samples, the segmentation network encodes visual embedding features. The GNN makes predictions by classifying and mapping these visual features to labels. The loss function is computed using formulas (4-6), and the loss value is backpropagated through the GNN model to update the parameters.

Q2: Concerns about using an older architecture and its competitive performance.

A: We appreciate your observations regarding the architecture used in our studies. To ensure a fair comparison, we selected this architecture based on the configuration outlined in Mseg [23]. Training on seven datasets is indeed time-consuming (approximately one week on four 80G A100 GPUs), which limits our capability to evaluate more competitive architectures at this stage. However, our results (Table 11) from five datasets show potential for improvement, as we've enhanced Cityscapes' mIoU to 82.2%. This indicates that there is room for further optimization in our method. We will explore more competitive base architectures, such as transformer-based models, in our future research.

Q3: Performance evaluation relative to the MSeg approach on indoor/outdoor datasets.

A: Thank you for raising this question. It's worth noting that MSeg combines categories and overlooks some difficult classes, which simplifies the learning task. Many of these difficult classes correspond to IoU values lower than the overall mean IoU (e.g., for ADE, mIoU for omitted/merged classes is 37.5 vs. all categories at 42.0; for COCO, it's 37.1 vs. 46.7). MSeg's merged categories also take into account label alignment across different datasets, which eases the learning process and thus may not represent a fair comparison to our method. We have provided examples of merged categories and their IoUs to help illustrate this point:

Q4: Potential issues with hallucinated label descriptions through ChatGPT.

A: Thank you for your invaluable input regarding the potential inaccuracies in generated label descriptions. We agree that using official label descriptions from datasets like Cityscapes greatly enhances the reliability of our methodology. Given that some datasets may lack corresponding label descriptions or have inconsistent language styles, this can pose challenges for the model. Therefore, in the future, we plan to use the label descriptions provided by official datasets as prompts, employing GPT models to generate more accurate and stylistically consistent label descriptions to help the model learn better. This adjustment may help improve the model's learning process by reducing the risks associated with hallucinated descriptions.

Limitation: Discussion on societal impacts and scalability challenges.

A: Thank you for the addition to the limitations section. Indeed, compared to unsupervised or weakly supervised methods, we still require complete annotated data. However, our approach leverages the available annotation information in existing datasets to reduce the reliance on annotated data. Errors in the fully automated construction of a unified label space do present some safety risks for autonomous driving tasks. Therefore, we also recommend introducing a manual review mechanism to address these issues. Current automatic label generation methods may introduce significant safety risks, so we recommend incorporating a manual review mechanism for generated labels to ensure accuracy and mitigate these concerns.

Regarding scalability, we concur that it's not necessary to load all datasets on a single node. Instead, as long as the computations for weight gradient updates include representations from all datasets, we can distribute this across multiple nodes, allowing more efficient processing. This approach can help facilitate scaling while avoiding bottlenecks in performance. We will include additional descriptions of these limitations in the final version of the paper, particularly focusing on safety considerations for autonomous driving scenarios.

We hope that these responses provide the clarity you were seeking and address your concerns adequately. We sincerely appreciate your constructive feedback, which is invaluable in improving our manuscript.

Dear Reviewer,

Thank you for your thoughtful feedback. We acknowledge the limitation of our current approach, where label descriptions generated by GPT are used to construct node features purely based on textual information. In future work, we plan to address this limitation by incorporating official label descriptions and introducing mask image features corresponding to the label categories to enhance the node features.

Regarding your question about MSeg, we would like to clarify that MSeg performs label merging or omission for certain datasets (e.g., ADE, COCO), while for others (e.g., CS, SUN), it uses the same evaluation categories as we do. For these datasets where a different evaluation protocol was used, while direct comparisons are not possible, this does explain the higher mIoU values observed in their results. However, we believe that on datasets where the same evaluation protocol is used, a fair comparison can be made.

Thank you again for your detailed feedback and for maintaining your positive view of our work.

We sincerely appreciate your thoughtful inquiries and for pointing out crucial elements of our methodology. Your feedback is instrumental in refining our paper and addressing any weaknesses. Below, we summarize your questions and provide detailed responses.

Q1: About the creation of textual descriptions using ChatGPT.

A: Thank you for pointing this out. We apologize for not providing specific details in our initial submission. In the revised version of the paper, we will clarify that we used ChatGPT 3.5 to generate label descriptions without any image input. Each dataset label category was formatted into the text input as “An image of <label> from the dataset <dataset>.” This prompt was used to encourage ChatGPT to complete the label descriptions. The resulting complete label descriptions were then inputted into a text encoder to generate the corresponding textual features for the label nodes.

Q2: Where is the final resulting unified label mapping?

A: Please refer to our global rebuttal for more examples. We plan to present our complete unified label mapping in the form of open-source code.

Q3: The importance of Supplement B regarding unified labels and the choice of hyperparameter Lambda |L|.

A: We appreciate you highlighting the importance of this aspect. Based on experimental insights and referencing paper [52], we selected the hyperparameter ($\lambda$ = 0.5), as this setting provided a good initial value. In our experiments with seven training datasets (CS, MPL, SUN, BDD, IDD, ADE, COCO), we had a total of 448 label categories. With the selected parameter, the resulting unified label space $|L|$ comprised 231 labels. We will include these details in the revised version of the paper.

Q4: What pre-training was used for the initialization of the multi-head models?

A: In the Multi-SegHead Training Stage section of Supplementary A, we describe the process of training our multi-head model. Specifically, we construct a separate segmentation head for each dataset, where each head contains only a linear layer to map the embedded feature channels to the specific label categories of that dataset. The training was conducted using four 80G A100 GPUs for 100K iterations. Image preprocessing and hyperparameter settings were consistent with those used in other training stages.

Q5: Treatment of WD2 negative examples in benchmarking datasets.

A: We apologize for the lack of clarity in our description, which may have caused confusion. We adhere strictly to the WD2 evaluation guidelines, applying uniform label mapping to all samples. This process is indeed applied to both the positive and negative frames. We will explicitly include this description in the revised version of the paper.

---

Weakness 1: Multi-head training and dataset bias concerns.

A: Thank you for highlighting the issue of dataset bias in multi-head training. We acknowledge this limitation. After completing cross-validation, we discard the multi-heads and the model utilizes a unified label embedding space for predicting class probabilities during subsequent training phases, which should help mitigate biases introduced by the initial training setup. We believe adding more datasets may help alleviate out-of-distribution label concerns, as combining datasets may reduce errors from mislabeling during cross-validation. We also recognize the importance of distinct semantic gaps, such as between "tunnel" and "fireplace". We plan further research to leverage label descriptions from official datasets and utilize advanced large models to provide accurate label mappings while constraining the model’s output based on textual features to minimize incorrect connections between semantically different classes.

Weakness 2: Splitting of source labels with respect to multi-head training.

A: We apologize for any confusion regarding the isolation of multi-heads training. After the initial multi-seghead training stage, we discard the multi-head structure, and thus, only a single UniSeghead is maintained for predictions. During inference on unseen images, we solely output results from the unified label space, eliminating the complications related to the isolation of respective heads. This will be clarified in the revised version of the paper. Additionally, our visualization results (in Supplement F) show that the final model generates fine-grained segmentation results.

Weakness 3 & 4: The use of both WD1 and WD2, and unclear figures or descriptions.

A: Thank you for noting the potential confusion arising from using both WD1 and WD2. To clarify, our model did not utilize WD1 during training, and we plan to remove references to testing on WD1 in the final version of the paper for better understanding. We also appreciate your reminder regarding the citation of the WD2 paper, which we will include in the revised version of the paper. Additionally, thank you for pointing out the lack of model descriptions in our visual results that caused confusion. The L223 experiment in the paper refers to the 7ds training model, which does not include WD2, resulting in only MPL having lane markings, crosswalks, and manholes. We will add more detailed descriptions to the images (Fig. 3 and Fig. 5) and revise 'Ours' to 'Our 7ds model'."

We hope these responses provide clarity and address your concerns effectively. Thank you once again for your constructive feedback.

We sincerely thank for your thoughtful questions. Below are our responses to the specific questions raised:

Q1: How do you select the unified label size N? Is it adaptive to different dataset labels, or preset hyper-parameters?

A: We apologize for not clearly explaining the selection of the unified label size N in the main text, which might have caused some confusion. In Supplementary Material B, we clarify that N is determined by solving Algorithm 2, rather than being a preset hyper-parameter. Specifically, we construct the label co-occurrence relationships through cross-validation of the multi-seghead model across different datasets. This automatic solving method allows us to adapt to different dataset labels and enhances scalability. We will add this explanation to Section 3.2 of the main text in the revised version of the paper.

Q2: Can you show more results of the bipartite graph generated by the GNN? Like what labels are hard to link by text features, but linked by the GNN.

A: Please refer to our global rebuttal for more examples. For instance, ADE: Signboard is annotated as both Traffic sign (front) and Traffic sign (back), while CS annotates Traffic sign (back) as an ignore category. We believe that such knowledge should be learned through samples.

Q3: Is the generated graph by GNN influenced by the number of datasets in training or the models? For example, if I use different models for the training, will the graph be the same? If I use two datasets in multi-dataset training, will the nodes of these two dataset labels have the same edges as when I use three datasets?

A: Yes, the selection of different models and datasets affects the constructed nodes. The GNN relies on the model’s segmentation predictions to learn the label mapping matrix. If the model cannot accurately recognize fine-grained categories, the GNN will not be able to construct the label mappings for these fine-grained categories. The influence of datasets on the generated graph is likely present. For instance, in 5ds training, IDD: Non-drivable fallback or rail track is merged with Terrain. However, in 7ds training, after removing WD2: Terrain and adding COCO: Railroad, IDD: Non-drivable fallback or rail track is merged with the fine-grained categories MPL: Rail Track and COCO: Railroad. We speculate this is because the inclusion of COCO: Railroad in 7ds training improves the model’s learning of fine-grained categories, affecting the prediction results for Non-drivable fallback or rail track in the IDD dataset, leading to different outcomes.

Q4: Can I apply the generated graph to other models' training without the need for doing GNN training again? If not, how much training cost will GNN bring?

A: Yes, similar to other multi-dataset training methods that apply manually designed unified label spaces to different models, our GNN model constructs a unified label space as a boolean label mapping matrix that can be separated and used for training other models. We will open-source the constructed unified label space along with the revised version of our code to facilitate training with other models.

If there are any further questions or concerns, please feel free to reach out to us.

Dear Reviewer,

thank you very much for your positive feedback. We acknowledge your concern regarding the hallucinated descriptions of the labels generated by GPT-3.5, particularly the potential shortcomings of relying on extended text without incorporating the images or masks themselves. We agree that integrating the corresponding image masks into the node features could enhance the accuracy of label generation and help mitigate the issues related to erroneous outputs from the model. Indeed, this is an important consideration, and we plan to explore this direction in our future research.

Thanks!