Response to Reviewer fVFW

We sincerely appreciate your volunteered time. We will respond to your concerns and hopefully eliminate them.

• R1 (About comparing distributions):

  As you mentioned, we compare the fitting ability of several candidate distributions in describing the feature space from a fully supervised learning perspective in Section 3.2 of the manuscript. Based on your valuable suggestions, we impose a comparison experiment in the distribution alignment branch of DGNet for several distributions to enhance the credibility of the article. The relevant experimental results are reported in Table 3 of rebuttal.pdf.

  We make the following modifications to embed the other three distributions individually into the distribution alignment branch of DGNet. (1) The Category Prototype models cluster features by distance magnitude, so we could not design a loss derived from maximum likelihood estimation like $\mathcal{L}\_\text{vMF}$ and the total loss used for Category Prototype models is $\mathcal{L}\_\text{tCE}+\mathcal{L}\_\text{DIS}+\mathcal{L}\_\text{CON}$. Correspondingly, there is no EM algorithm for optimizing the parameters in the distribution alignment branch. (2) For the Gaussian Mixture Model, we modify the Gaussian Mixture Model loss [1] used for images into the distribution alignment branch of DGNet.

  The experimental results show that the segmentation performance of different distributions under weak supervision is consistent with the feature space fitting ability under full supervision. The stronger feature space fitting ability of moVMF under full supervision leads to more accurate and effective supervised signals for weakly supervised learning with the distribution alignment branch in DGNet.

• R2 (About typos and minor flaws):

  Thanks for the meticulous reading! The typos and minor flaws you mentioned will be rectified in the next release.

• R3 (About balance weights):

  As mentioned in line 221 of the text, instead of adjusting the weight of each loss term, DGNet significantly improves the network's performance by simply adding them together, reflecting the robust performance improvement from each loss term. Recognizing that careful tuning of the weights may lead to superior segmentation performance, we fix the $\mathcal{L}\_\text{tCE}$ and balance the weights of other loss terms by magnitude, the results of which are shown in Tab. 1 of Rebuttal.pdf. The balanced loss function is $\mathcal{L}\_\text{tCE}+\mathcal{L}\_\text{vMF}+0.1\mathcal{L}\_\text{DIS}+0.1\mathcal{L}\_\text{CON}$, but essentially equal in performance to the loss function used in the manuscript.

• R4 (About outdoor datasets):

  Only a few works provide segment performance on outdoor datasets in weakly supervised point-cloud semantic segmentation. With the same baseline method RandLa-Net, DGNet improves 1.0% and 2.1% mIoU over SQN on SemanticKITTI at 0.1% label rate and 0.01% label rate, respectively. Considering the inherent density inhomogeneity of outdoor scenes, the excellent performance with such sparse annotations validates the effectiveness of DGNet with the outdoor dataset.

  To enrich the evaluation of the outdoor dataset, we compare the segmentation performance of the baseline method RandLA-Net and DGNet on the Semantic8 subset of Semantic3D. At 0.01% label rate, DGNet achieves 59.4% mIoU compared to 55.8% of the baseline.

[1] Wu, Linshan, et al. "Sparsely annotated semantic segmentation with adaptive gaussian mixtures." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Dear Reviewer fVFW:

Given that the deadline for the reviewer-author discussion period is less than two days away, we are eager to discuss our work with you to clear up any concerns you may have. Overall, we have made the main efforts based on your comments as follows:

• One comparison experiment in the distribution alignment branch of DGNet for several distributions.

• One weight balance experiment for tuning the weights of Eq. 9.

• One detailed discussion of DGNet in outdoor scenarios.

We understand your time is valuable, but your opinion updates are vital to our work.

Response to Reviewer ruvS

We sincerely appreciate your insightful feedback and positive evaluation. In response to your valuable comments, we structure our response as follows:

• R1 (About oughtness and moVMF):

  Next, we discuss "oughtness" and the superiority of the mixture of vMFs (moVMF).

  • Discussion about "oughtness".

    In DGNet, the "oughtness" is a constraint on the feature space distribution under weakly supervised learning. The goal of weakly supervised learning is to enable the network to achieve comparable performance under sparse annotations as under full annotations. To achieve this, the features under weakly supervised learning should be as identical as possible to those under fully supervised learning. Due to the great discrepancy in the quality of the annotations between weakly supervised learning and fully supervised learning, obtaining the same features is difficult. Therefore, in this paper, we relax this restriction and expect to realize the alignment of fully supervised learning and weakly supervised learning in the feature space of the neural network. The "oughtness" means that the feature space distribution under weakly supervised learning should be identical to the feature space distribution under fully supervised learning.

  • The superiority of moVMF.

    Above we discuss the "oughtness" of weakly supervised learning, but how to describe the feature space distribution under fully supervised learning to provide a prior for weakly supervised learning has not been addressed. Unfortunately, how to describe the feature space of a neural network is an open problem. However, it is undeniable that von Mises-Fisher has demonstrated strong data fitting and generalization capabilities in the fields of self-supervised learning [1,2], classification [3], variational inference [4], online continual learning [5], and so on. For our explanation of the superiority of moVMF, please refer to the R3 (About discussion and analysis of moVMF) in the response to the Reviewer KHTz.

• R2 (About representation ability):

  Firstly, normalized features only as an input to the distribution alignment branch to provide additional supervised signals to DGNet, while the weakly supervised learning branch uses unnormalized features, and thus for the theoretical upper bound of the representation, DGNet does not differ from the baseline. Second, for the D-dimensional semantic feature $\mathbf{F}$, the normalized feature $\text{norm}(\mathbf{F})$ in the distribution alignment branch is a feature distributed on a D-1 dimensional hypersphere. Given that D is typically large (set to 256 in our implementation), the loss due to projection is almost negligible. Furthermore, we know from previous analyses that the segment head is a radial classifier, so the normalized features do not affect the segmentation results either.

• R3 (About performance on full supervision):

  The experiment in Table 1 is a validation of the neural network feature distribution under full supervision. The experimental results illustrate that the feature space distribution under full supervision fits more closely to the mixture of von Mises-Fisher distribution than the other distributions. Following your suggestion, we impose our DGNet to full supervision. With PointNeXt-L as the baseline, DGNet slightly improves mIoU from 69.2% (baseline) to 69.5% on S3DIS, which means that distribution guidance is similarly effective for full supervision.

• R4 (About typos and reference):

  Thanks for your meticulous reading! The sentence in Line 82 should be " Via CAMs, MPRM and J2D3D dynamically generate point-wise pseudo-labels from subcloud-level annotations and image-level annotations, respectively. " We will correct this typo in the next version. We also note that some encouraging work has emerged from CVPR 2024, but these articles were not available before submission. We will cite them in the next version.

[1] Chen, Xinlei, and Kaiming He. "Exploring simple siamese representation learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[2] Hariprasath Govindarajan, Per Sidén, Jacob Roll, Fredrik Lindsten. "DINO as a von Mises-Fisher mixture model." ICLR 2023.

[3] Scott, Tyler R., Andrew C. Gallagher, and Michael C. Mozer. "von mises-fisher loss: An exploration of embedding geometries for supervised learning." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[4] Taghia, Jalil, Zhanyu Ma, and Arne Leijon. "Bayesian estimation of the von-Mises Fisher mixture model with variational inference." IEEE transactions on pattern analysis and machine intelligence 36.9 (2014): 1701-1715.

[5] Michel, Nicolas, et al. "Learning Representations on the Unit Sphere: Investigating Angular Gaussian and Von Mises-Fisher Distributions for Online Continual Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 13. 2024.

Thanks for your prompt response! We are pleased that our response addresses most of your concerns. We also respect the importance you place on von Mises-Fisher distribution, and we will further elaborate on its superiority in solving the weakly supervised point cloud semantic segmentation task from multiple perspectives in our next release.

We consider further analyzing the difference between the Euclidean norm (Gaussian mixture model) and cosine similarity (moVMF) from the perspective of the Curse of Dimensionality.

• The data becomes extremely sparse in feature space as the feature dimension increases. Most feature vectors are far from each other, causing the Euclidean distance to become ineffective in distinguishing differences between feature vectors. Cosine similarity, on the other hand, is more effective in distinguishing differences between features by measuring the angle between the vectors.

• Euclidean norm is very sensitive to scale. In a high-dimensional feature space, if the feature changes at different scales in different dimensions, it may result in the contribution of some dimensions to the Euclidean norm being magnified while the contribution of other dimensions is ignored. Cosine similarity is not affected by the length of the vectors, and the main measure is the angle between the features. Therefore, changes in the scale of different dimensions do not significantly affect the cosine similarity calculation.

Response to Reviewer bWaK

We appreciate your valuable comments and suggestions. We will respond to each of your concerns and hopefully eliminate them.

• R1 (About detailed explanation):

  Due to page constraints, we analyze and interpret the main and important experimental results in the manuscript. We will provide a detailed elaboration of the analysis of the experimental results in Sec. 4.2 and 4.3 during the reviewer-author discussion period, due to the character limit in the rebuttal period.

• R2 (About writing):

  In our writing experience, there are generally three main types of writing in academic articles:

  • The article proposes a new task, which requires a description of the significance of the new task and the difficulties in solving it.
  • The article provides specific improvements to the existing methodology, which needs to describe the problems faced by the methodology.
  • The article develops a new paradigm to address the current challenges, which entails summarizing the previous paradigms and elaborating on the advantages of the proposed new paradigm.

  We believe that DGNet is more per the third writing logic. Therefore, we focus on the challenges of weakly supervised point cloud semantic segmentation in the Abstract. In the second paragraph of the Introduction, we summarize current methods into three paradigms and draw out their common problem, for which we propose a new paradigm.

• R3 (About contribution summary):

  According to our understanding, NeurIPS 2024 does not have a mandatory requirement to summarize contributions. Some highly cited papers also do not summarize major contributions, such as ResNet, Attention is All You Need, Non-local Neural Networks, ViT, etc.

• R4 (About balance weights):

  As mentioned in line 221 of the text, instead of adjusting the weight of each loss term, DGNet significantly improves the network's performance by simply adding them together, reflecting the robust performance improvement from each loss term. Recognizing that careful tuning of the weights may lead to superior segmentation performance, we fix the $\mathcal{L}\_\text{tCE}$ and balance the weights of other loss terms by magnitude, the results of which are shown in Tab. 1 of Rebuttal.pdf. The balanced loss function is $\mathcal{L}\_\text{tCE}+\mathcal{L}\_\text{vMF}+0.1\mathcal{L}\_\text{DIS}+0.1\mathcal{L}\_\text{CON}$, but essentially equal in performance to the loss function used in the manuscript.

• R5 (About qualitative comparison):

  Our work follows the visualization experimental setup of state-of-the-art methods (such as ERDA, CPCM, PointMatch, etc) in the field of weakly supervised point cloud semantic segmentation, focusing on visual comparisons between the baseline and the proposed method.

• R6 & R7 (About performance improvement):

  For a detailed explanation of performance, including why DGNet has less boost in some cases, please refer to the specific response in R1.

  We would like to emphasize that DGNet is not an incremental work, i.e., it is not an adaptation or refinement of an existing weakly supervised learning paradigm, but rather a new weakly supervised learning paradigm constructed from the perspective of describing the distribution of feature space. Therefore, the purpose of the comparison experiments is to demonstrate that this novel weakly supervised learning paradigm is feasible and has the potential to outperform other weakly supervised learning paradigms. We recognize that there is still room for performance improvement in some datasets, but we believe the current comprehensive performance benefits across multiple datasets and multiple sparse labeling settings are sufficient to demonstrate the potential of this new paradigm. We look forward to the subsequent ongoing efforts of the Point Cloud community to fully exploit the weakly supervised performance of this paradigm.

• R8 (About performance degradation and resource consumption):

  • Explanation for performance degradation We find two places that could be considered performance degradation in Sec. 4.3. The first place is the introduction of $\mathcal{L}\_\text{vMF}$ with hard assignment form. In contrast to the soft assignment, the hard assignment does not take into account the inter-cluster similarity and is mismatched with the soft-moVMF algorithm. So DGNet uses $\mathcal{L}\_\text{vMF}$ with soft assignment form. The second place is to optimize $\alpha$ and $\mathbf{u}$ individually because the fixed parameters affect the accurate update of the optimizable parameters. Therefore, DGNet simultaneously optimizes $\alpha$ and $\mathbf{u}$.
  • Explanation for resource consumption In the inference phase, there is no additional memory consumption compared to the baseline since DGNet only activates the weakly supervised learning branch for inference. In the training phase, the resource consumption mainly comes from the Memory Bank and Soft-moVMF algorithm. Compared to the baseline, DGNet only increases 25.8M during the training phase, and this resource consumption increase is not significant compared to the baseline network itself.

Next, we provide a detailed elaboration of the analysis of the experimental results in Sec. 4.2 and 4.3. Please note that these additions of details do not detract from the main conclusions in the paper, and we believe that some details are so burdensome that may dilute the focus on key experiments and make the paper lengthy.

• Results on S3DIS. The gain that DGNet brings to the baseline method is more pronounced at 0.01% label rate compared to 0.1% label rate, since the guidance on feature distribution is more valuable with extremely sparse annotations. The 0.02% label rate denotes a sparse labeling form of "one-thing-one-click", and unlike methods that use this labeling form, we did not introduce super-voxel information to enrich the original sparse labeling. However, DGNet outperforms these methods at a smaller label rate. In addition, we would like to re-emphasize that DGNet demonstrates strong generalization capabilities, both in terms of different annotation scales and various baseline methods.

• Results on ScanNetV2. Confronted with the diverse categories and versatile scenes of ScanNetV2, we follow the other methods and report on the segmentation performance with the sparse annotations processed by super-voxels. The cross-entropy loss term can generate relatively sufficient supervised information to train the network due to the introduction of pseudo-labelings, resulting in a less pronounced improvement of DGNet than that of S3DIS. However, DGNet is still slightly superior to the latest SOTA methods.

• Results on SemanticKITTI. In the field of weakly supervised point-cloud semantic segmentation, only a few works provide segment performance on outdoor SemanticKITTI. With the same baseline method RandLa-Net, DGNet improves 1.0% and 2.1% mIoU over SQN at 0.1% label rate and 0.01% label rate, respectively. Considering the inherent density inhomogeneity of outdoor scenes, the excellent performance with such sparse annotations validates the effectiveness of DGNet with the outdoor dataset.

• Interpretability of prediction results. The interpretation of Bayesian posterior probabilities for trained networks based on the distribution function of the moVMF is an attractive advantage of DGNet. Figure 3 visualizes the posterior probabilities for some categories. Taking the “floor” as an example, according to the Bayesian theorem, those points with relatively high posterior probabilities have high probabilities of belonging to the floor, which is also consistent with the network predictions.

• Hyperparameter selection. In Table 5, we search the parameter space for suitable $\kappa$, $t$, and $\beta$. We observe that (1) the segmentation performance shows an increasing and then decreasing trend as the concentration constant $\kappa$ increases. Our analysis suggests that too small $\kappa$ leads to a dispersion of features within the class, which can be easily confused with other classes. And too large $\kappa$ forces overconcentration of features within the class and overfits the network. (2) As the iteration number $t$ increases, the segmentation performance gradually rises and then stabilizes. We believe that the soft-moVMF algorithm gradually converges as $t$ increases, and increasing $t$ after convergence will no longer bring further gains to the network. (3) As the truncated threshold $\beta$ decreases, the segmentation performance shows a tendency to first increase and then decrease. The conventional cross-entropy loss function is the truncated cross-entropy loss function with $\beta=1$. When $\beta$ decreases, the overfitting on sparse annotations is alleviated, but when $\beta$ is too small, it weakens the supervised signal on sparse labeling leading to performance degradation.

• Ablation study for loss terms. Table 6 demonstrates the validity of each loss term in DGNet. It is worth noting that $\mathcal{L}\_\text{vMF}$ with hard assignment form undermines the segmentation efficiency, on the contrary, $\mathcal{L}\_\text{vMF}$ with soft assignment form has a clear positive effect. In contrast to the soft assignment, the hard assignment does not take into account the inter-cluster similarity and is mismatched with the soft-moVMF algorithm. In addition, according to suggestions from another reviewer, we supplement this ablation experiment with more ablative forms. Please refer to the response to the Reviewer KHTz.

• Ablation study for soft-moVMF. The experimental results in the first, second, third, and last rows of Table 7 show that it is optimal to optimize both $\alpha$ and $\mathbf{u}$ with the soft-moVMF algorithm. Optimizing $\alpha$ and $\mathbf{u}$ individually impairs the segmentation performance, for which the fixed parameters affect the accurate update of the optimizable parameters. The forth, fifth and last rows of Table 7 show that the soft-moVMF outperforms hard-moVMF and KNN-moVMF due to more accurate parameter updates.

Dear Reviewer bWaK:

Given that the deadline for the reviewer-author discussion period is less than two days away, we are eager to discuss our work with you to clear up any concerns you may have. Overall, we have made the main efforts based on your comments as follows:

• Further explanation of the experimental results.

• One weight balance experiment for tuning the weights of Eq. 9.

• Detailed clarification of the article writing.

We understand your time is valuable, but your opinion updates are vital to our work.

Response to Reviewer KHTz

We express our sincere gratitude for your valuable comments. In response to your concerns, we carefully structure our response as follows:

• R1 (About computational complexity):

  The main computational complexity of the distribution alignment branch comes from the soft-voVMF algorithm. According to the pseudocode in the Supplementary Material, the complexity of the soft-voVMF is $O(tn|\mathbb{C}|)$, where $t$ is the iteration number, $n$ is the point number of the point cloud, and $|\mathbb{C}|$ is the number of semantic categories. Since $t$, $n$, and $|\mathbb{C}|$ are all set to constant values during network training, the extra computation introduced by the distribution alignment branch is trivial. A detailed explanation of why the algorithm converges quickly at a constant $t$ is given in R3.

• R2 (About ablation study):

  Table 6 of the manuscript illustrates the ablation experiments with the loss term added individually. Based on your valuable suggestion, we show the ablation study for removing the loss terms individually in the rebuttal.pdf. As can be seen, each loss term contributes to the final result. Removing or modifying loss terms results in sub-optimal performance, where vMF loss has the most significant effect.

• R3 (About discussion and analysis of moVMF):

  • Why moVMF?

    How to describe the feature space of a neural network is an open problem. Since the feature space is affected by a variety of factors such as network structure, training data, parameter settings, optimization (loss) function, etc., it is unrealistic to mathematically derive an optimal and general description. Nevertheless, we construct different distributions from two dimensions and try to explain the merits and demerits of these candidate distributions in describing the feature space.

    • From the distribution modeling perspective, we compare the Category Prototype Model with the Mixture Model. Despite its computational simplicity, the Category Prototype Model describes the features only by comparing the distance of features to the category prototypes. This means that the Category Prototype Model ignores the distribution within categories and the variability between categories. In contrast, the Mixture Model has a larger parameter optimization space and a relatively stronger fitting ability.

    • For the distance metric, we compare the Euclidean norm and cosine similarity. We attempt to analyze these two distances with the segment head structure in the Supplementary Material. To facilitate the analysis, we simplify the structure of the segment head and define it as $\text{SegHead}(\mathbf{F})=\text{argmax}(\text{softmax}(\mathbf{WF}^\top))$, where $\mathbf{F}$ is the semantic feature extracted by the decoder and $\mathbf{W}$ is the parameter of the output layer. Consider a group of feature vectors {$k\mathbf{F} |k\geq 0 \And \mathbf{F} \neq \mathbf{0}$}. For any two feature vectors $k_1\mathbf{F}$ and $k_2\mathbf{F}$ within this group, the segmentation predictions are identical, $\text{SegHead}(k_1\mathbf{F})=\text{argmax}(\text{softmax}(k_1\mathbf{WF}^\top))=\mathop{\text{argmax}}\limits_{i} \frac{e^{k_1 \mathbf{W}_{i,:} \mathbf{F}\_{i,:}^\top } }{\sum_j e^{k_1 \mathbf{W}\_{j,:}\mathbf{F}\_{j,:}^\top}} = \text{argmax}(\text{softmax}(\mathbf{WF}^\top))=\text{argmax}(\text{softmax}(k_2\mathbf{WF}^\top))=\text{SegHead}(k_2\mathbf{F}).$ If the general case of using an activation function is taken into account, it does not change the result after argmax since the activation function is usually monotonically nondecreasing. Therefore, the segment head is a radial classifier that has a more pronounced classification performance on the angles, so that cosine similarity describes the feature space better than the Euclidean norm.

    In summary, the mixture of von Mises-Fisher distribution with both cosine similarity and mixture model is optimal among the candidate distributions, and the experimental comparisons in Tab 1 validate our analysis.

  • Convergence proof.

    The soft-moVMF belongs to the EM algorithm. In 1983, Jeff Wu proved the convergence properties of the EM algorithm in [1]. He proved that the EM algorithm can make the log-likelihood function of the observed data converge to a stable value under certain conditions.

    However, this stable value is not necessarily optimal, which is related to the initialization of the EM algorithm. Therefore, we prove that the difference between the initialization $\mathbf{h_c}$ from the weakly supervised learning branch and the theoretical optimal mean vector $\mathbf{u^*\_c}$ conforms to a Gaussian distribution with a mean value of 0. According to maximum likelihood estimation, $\mathbf{u}^*\_c=\frac{1}{n_c}\mathop{\sum}\limits_{y\_i = c} (\mathbf{F}\_i)$, where $n_c$ denotes the point number belonging to c-th category and $y_i$ denotes the i-th label of full annotations. The initialization $\mathbf{h_c}$ from the weakly supervised learning branch can be defined as $\mathbf{h_c}=\frac{1}{m_c}\mathop{\sum}\limits_{y'_i = c} (\mathbf{F}_i)$, where $m_c$ is the labeled point number belonging to c-th category in sparse annotations and $y'_i$ is the i-th label of sparse annotations. Sparse labeling can be considered as the result of sampling over the full labeling, i.e., $y'=s(y)$. Assume that the sample is independently redistributed and obeys a certain distribution $\mathcal{D}(\mu,\sigma^2)$. According to the Central Limit Theorem, the following convergent distribution can be obtained: $(\mathbf{u}^*\_c-\mathbf{h}\_c) \sim \mathcal{N}(0, \frac{\sigma^2}{m_c}).$ This means that the initialized mean vector in DGNet has a high probability of appearing in the vicinity of the optimal solution. Therefore, the soft-moVMF converges quickly at a constant iteration number $t$.

[1] Wu, CF Jeff. "On the convergence properties of the EM algorithm." The Annals of statistics (1983): 95-103.

Dear Reviewer KHTz:

Given that the deadline for the reviewer-author discussion period is less than two days away, we are eager to discuss our work with you to clear up any concerns you may have. Overall, we have made the main efforts based on your comments as follows:

• A crucial discussion on the computational complexity.

• An ablation study for removing the loss terms individually.

• The discussion and analysis of the superiority of moVMF.

We understand your time is valuable, but your opinion updates are vital to our work.