WI3D: Weakly Incremental 3D Detection via Visual Prompts

[To weakness5, "more learning stages and more classes"] Thank you for your valuable suggestion. Due to time constraints, we conduct experiments specifically involving continual learning across 3 stages on ScanNet [R8], covering a total of 30 categories (in alphabetical order [R4]). It is evident that compared to SDCoT, our proposed method exhibits reduced susceptibility to catastrophic forgetting during sequential incremental learning. These experiments collectively demonstrate the superiority and robustness of our approach.

Table r1: Per-class performance (AP@0.25) of SDCoT and WI3D on ScanNet val set under the sequential incremental learning setting. B[1-10] denotes standard training on ten base classes. N[11-20] and N[21-30] denote sequential incremental training on the novel classes.

[To question1, "check the references"] Thank you for your correction. We have addressed this issue in the revised version.

[To question2, "experiment on outdoor datasets"] Compared to outdoor datasets, indoor scenes typically encompass a more diverse range of object categories, making them well-suited for incremental learning tasks. Previous works on class-incremental object detection focus solely on indoor scene datasets[R4-R6]. As a result, we conduct extensive experiments on two widely used indoor 3D datasets, SUN RGBD[R7] and ScanNet[R8], and compare our approach to traditional class-incremental learning methods. Due to significant differences in point cloud processing, model architecture, and training specifics between methods applied to indoor and outdoor datasets, we are presently unable to provide experimental results on outdoor datasets. Exploring weak incremental object detection under outdoor scenarios remains an area for future investigation.

Reference

[R4]Zhao N, Lee G H. Static-dynamic co-teaching for class-incremental 3d object detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(3): 3436-3445.

[R5]Zhao Z, Xu M, Qian P, et al. DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detection[J]. arXiv preprint arXiv:2212.02057, 2022.

[R6]Liang W, Sun G, Liu C, et al. I3DOD: Towards Incremental 3D Object Detection via Prompting[J]. arXiv preprint arXiv:2308.12512, 2023.

[R7]Song S, Lichtenberg S P, Xiao J. Sun rgb-d: A rgb-d scene understanding benchmark suite[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 567-576.

[R8]Dai A, Chang A X, Savva M, et al. Scannet: Richly-annotated 3d reconstructions of indoor scenes[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 5828-5839.

We appreciate your approval of our idea and the insightful comments. Your concerns will be addressed in the following comments. We have also updated the revision of our paper accordingly. 

[To weakness1, "generate the coarse pseudo labels"]  In our method, the 3D pseudo-labels are constructed in a way similar to that of Peng et al. (2021) [R1].To be specific, we first project the point cloud onto the image plane and select points within each 2D bounding box. Then, we adopt DBSCAN[R1](eps=0.25, min_samples=100) to divide the points in each 2D box into multiple clusters according to the distance and density between the points. After that, we drop the clusters which contain fewer points than 1/10 of the number of points in that 2D box, which ensures the generation of a 3D instance mask. In addition, we select the cluster with the largest cardinality and calculate a coarse 3D bounding box with PCA to estimate the rotation, center, and size of the 3D pseudo box. As this is not the primary focus of our innovation, we did not elaborate on it in the main paper. We have updated the details in the supplementary materials.

[To weakness1, "knowledge transfer"]  In section 3.4, we utilized an MLP-based projection layer to encode the 3D proposal features into the same feature space as the 2D region embedding. Similar to the representation of base-class objects, the representation of novel-class objects also receives direct supervision from pseudo labels. Additionally, prior works such as [R2] also propose knowledge transfer for novel-class objects for better learning of novel objects.

[To weakness2, "contribution"]  Our key contribution is that we first propose and explore the task Weakly Incremental 3D object Detection, which is an incremental 3D learning scheme in label-efficient scenarios. Because of the lack of novel-class 3D annotations and domain differences between images and point clouds, WI3D is a challenge and valuable task. To this end, we propose a robust and effective framework to deal with it, which contains a class-agnostic pseudo-label refinement module for high-quality pseudo-label generation and concept representation learning in feature space. Compared with the traditional workflow of 3D object detection, WI3D can further generalize the class-incremental 3D object detection to more realistic settings via cost-effective 2D visual prompts only. We earnestly aspire that our endeavors in the realm of weakly 3D class-incremental learning tasks will spark inspiration and accelerate future research in this community.

[To weakness3, "how does teacher predict reweighting modulation factor"] During base training, we predict an objectness score[R3], indicating the predicted confidence that the object belongs to the base classes. While continuously learning novel classes, we utilize the softmax function to compute the objectness score of the current proposal concerning the total confidence among all proposals, acting as the reweighting modulation factor. We have updated the formula for calculating the reweighting modulation factor in the manuscript.

[To weakness3, "fine-tuning” method] We apologize for the lack of clarity on this method. You're correct - we perform fine-tuning on the whole model(except the base classifier) and a new classifier for the Cnovel. We revised the description of the "fine-tuning" method in the manuscript.

[To weakness3, "vanilla method" in Table 5] In Table 5, "vanilla" denotes the baseline without PLR (details in sec. 3.3), CKT and RKD (details in sec. 3.4). "Our" refers to the results achieved by our pipeline shown in Figure 3. We included the description for “vanilla” in the revised paper.

[To weakness4, "effectiveness of PRF module"] Thank you for your valuable suggestions, we have provided results and analysis based on recall of coarse and refined pseudo labels in the supplementary material. The results demonstrate that PRF significantly improves the estimated bounding boxes of novel classes, bringing them closer to the ground truth labels.

Reference

[R1] Peng L, Yan S, Wu B, et al. Weakm3d: Towards weakly supervised monocular 3d object detection[J]. arXiv preprint arXiv:2203.08332, 2022.

[R2]Zhao Z, Xu M, Qian P, et al. DA-CIL: Towards Domain Adaptive Class-Incremental 3D Object Detection[J]. arXiv preprint arXiv:2212.02057, 2022.

[R3] Gupta A, Narayan S, Joseph K J, et al. Ow-detr: Open-world detection transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 9235-9244.

[To weakness2, "similarities with the BoxPC"]  Though BoxPC also refines pseudo labels for novel classes, our PLR differs from BoxPC in motivation, model design and settings.

Motivation: As outlined in sections 3.1 and 3.3, the process of projecting 2D bounding boxes into 3D space introduces several challenges, including “projection migration", "scale ambiguity", and "overlap". To address these issues, we introduce a class-agnostic Pseudo Label Refinement (PLR) module. By decoupling 3D boxes and learning pseudo-label offsets and scales, our approach aims to resolve the problems stemming from Projection Migration and Scale Ambiguity. Furthermore, an additional Binary Classification Header (BCH) is employed to mask duplicated estimations. The BoxPC network, however, focuses on transferring knowledge learned from strong classes to weak classes and does not analyze or address the arising noise.

Model Design: 1) BoxPC manually designs specific thresholds such as bounds α+,α−,β+,β−. In contrast, PLR does not require manual threshold setting. 2) BoxPC predicts a single fit probability p intended to constrain PointNet-based network training. In PLR, the Binary Classification Header (BCH) predicts the probability of presence compared to absence simultaneously, and the probability comparison helps mask duplicated estimations.

Settings: 1) BoxPC accepts only one 3D pseudo box and its corresponding local frustum point cloud as input. Conversely, multiple boxes and the whole scene are simultaneously fed in PLR, allowing it to utilize global and contextual information. 2) During training, BoxPC utilizes pseudo-labels generated by perturbing 3D ground truth labels as input. However, PRF matches the disordered pseudo-labels generated from 2D visual prompts with the ground truth while refines the pseudo-labels.

[To weakness2, "comparing with removing alpha in the ablation study"]  We greatly appreciate your valuable suggestions. We supplement the corresponding experiments in Table r1, comparing our RKD with the results obtained by removing $\alpha$. The results in Table r1 demonstrate the effectiveness of our RKD with $\alpha$.

Table r1: Effectiveness of reweighting knowledge distillation (RKD) for weakly incremental 3D object detection. We compare our proposed RKD with other commonly used knowledge distillation manner, as well as the method without $\alpha$. "-" denotes that no distillation technology is used.

[To weakness3, "results in the sequential incremental learning setting"]  We perform experiments under a sequential incremental learning scenario, where the novel classes are split into subsets and become available sequentially. We evaluate the per-class average precision (AP) when the novel classes are added in a sequential manner. The results are shown in Table r2. It can be observed that compared with SDCoT, the proposed method suffers less from catastrophically forgetting during sequential incremental learning. All those experiments demonstrate the superiority and robustness of our method. 

Table r2: Per-class performance (AP@0.25) of SDCoT and WI3D on SUN RGB-D val set under the sequential incremental learning setting. B[1-5] denotes standard training on five base classes. N[6-8] and N[9-10] denote sequential incremental training on the novel classes.

[To weakness4, "The paper would benefit from visualization and the validity of the example shown in Fig 2(a)"] Thank you for your valuable suggestions and we have provided visual examples illustrating bipartite matching results in the updated supplementary materials. Furthermore, the illustration in column a of Figure 2 aims to elucidate the Projection Migration (Section 3.1 of the manuscript), which causes the displacement of the positions of the 3D bounding boxes. Additionally, we have replaced Figure 2 with a more illustrative one.

We appreciate your approval of our idea and the detailed and insightful comments. Your concerns will be addressed in the following comments and the revision of our paper has been updated accordingly. 

[To weakness1, "the method for generating coarse pseudo"]  In our method, the 3D pseudo-labels are constructed in a way similar to that of Peng et al. (2021).To be specific, we first project the point cloud onto the image plane and select points within each 2D bounding box. Then, we adopt DBSCAN[R1](eps=0.25, min_samples=100) to divide the points in each 2D box into multiple clusters according to the distance and density between the points. After that, we drop the clusters which contain fewer points than 1/10 of the number of points in that 2D box, which ensures the generation of a 3D instance mask. In addition, we select the cluster with the largest cardinality and calculate a coarse 3D bounding box with PCA to estimate the rotation, center, and size of the 3D pseudo box. As this is not the primary focus of our innovation, we did not elaborate on it in the main paper. We have updated the details in the supplementary materials.

[To weakness1, "clarification on normalization step and training process of the PLR"] During the normalization process, the coordinates of the point cloud are restricted to the output range within [0, 1]. Afterward, we normalize the center coordinates and dimensions of the pseudo boxes to fit within the corresponding range. Furthermore, the PLR module is trained alongside the 3D detection backbone in stage 1, due to their shared input (point cloud) and supervision (ground truth labels for base class).

[To weakness1, "modify the training of SDCoT to fit our weakly incremental learning setting"]  We adaptively modify SDCoT to fit our WI3D setting. Specifically, we introduce the process of coarse pseudo-label generation to SDCoT to obtain annotations for novel classes, instead of using novel-class ground truth. Other training processes are identical to SDCoT. For a fair comparison, all the training settings, e.g., learning rate, optimizer, batch size, etc., are the same for all experiments. 

[To weakness1, "details of the fine-tuning and freeze-and-add setup"]  "Fine-tuning" refers to first train the model on base classes, followed by fine-tuning the whole model as well as a new classifier for $C_{novel}$ except the classification heads for $C_{base}$. "Freeze-and-Add" is first trained on base classes, followed by adding a new classification head and fine-tuning only the new heads for $C_{novel}$. 

[To weakness1, "splitting the category set into C_base and C_novel according to SDCoT"]   Following SDCoT, we also take a subset of classes (such as 7 base classes / 3 novel classes, 5 base classes / 5 novel classes in table 1) in alphabetical order from each dataset as $C_{base}$ and treat the remaining as $C_{novel}$. Additionally, we compare baseline methods, SDCoT, and our method under the same class partitioning.

[To weakness1, "the term "vanilla" in Table 5 needs clarification. "]  In Table 5, "vanilla" denotes the our method without PLR (details in sec. 3.3), CKT and RKD (details in sec. 3.4). "Our" refers to the results achieved by our pipeline shown in Figure 3. We included the description for “vanilla” in the revised paper.

Reference

[R1] Ester M, Kriegel H P, Sander J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise[C]//kdd. 1996, 96(34): 226-231.

[To question7, "regional representations"] "regional representations" refer to the visual feature of a specific region in the 2D image extracted by a 2D teacher. We have updated the manuscript and specified its meaning.

[To question8, "the cost function"] We represent the 2D predictions and projected 3D proposals as two distinct sets. To facilitate optimal matching between these two sets—finding a corresponding 3D proposal for each 2D prediction—the bipartite matching method necessitates a cost function, which can be formulated as follows:

$min \sum_i \sum_j m_{ij} * IoU(Project(B_{i}^{3D}),{B}_{j}^{2D})$

$\sum_i m_{ij}=1$

where $m_{ij} \in {0, 1}$ indicates whether it matches, and $IoU$ represents computing the intersection over union. We update the formula and explanation for the cost function in Section 3.4. Furthermore, the cost function mentioned above is not part of the training objectives.

[To question9, "introduce, Pseudo, predictions, objectness"] Thank you for the notifications. We have fixed the typos accordingly.

Thank you for your valuable feedback. Your concerns will be addressed in the following comments and we have also updated the revision accordingly. 

[To weakness, "only one baseline in the evaluation"] The existing works include SDOCT (Zhao & Lee, 2022), DA-CIL (Zhao et al., 2022), and I3DOD (Liang et al., 2023). Among them, 1) Zhao & Lee (2022) introduce SDOCT for class-incremental 3D object detection which is open-sourced. In our evaluation, we assess SDCoT (Zhao & Lee, 2022) as the baseline model under the WI3D setting. 2) The official codebase of DA-CIL (Zhao et al., 2022) is currently unavailable. 3) Additionally, I3DOD (Liang et al., 2023), whose pre-print was initially released in August and was considered concurrent work, does not have code available to fit our settings.

[To question1, "the connection with the human visual system"] As mentioned in [R1], humans can learn new visual concepts from a broad range of data modalities (cross-modal). In our case, we consider the 2D visual prompts as the cross-modal data. Perceiving a 3D environment through image is akin to viewing the world through human eyes, including constantly learning from images, and identifying corresponding objects in the 3D-physical world.

[To question1, "what it means for visual prompts to be"] We utilize predictions (such as object‘s category, location, and features) generated by cost-free 2D teachers as "visual prompts". Unlike traditional class-incremental 3D object detection, we do not need delicate 3D annotations of novel-class objects, reducing the cost of 3D incremental learning.

[To question2 and question4, "the method used to generate the 3D boxes"] In our method, the 3D pseudo-labels are constructed in a way similar to that of Peng et al. (2021).To be specific, we first project the point cloud onto the image plane and select points within each 2D bounding box. Then, we adopt DBSCAN[R2] to divide the points in each 2D box into multiple clusters according to the distance and density between the points. After that, we drop the clusters which contain fewer points than 1/10 of the number of points in that 2D box, which ensures the generation of a 3D instance mask. Finally, this process selects the cluster with the largest cardinality and calculates a coarse 3D bounding box with PCA to estimate the rotation, center, and size of the 3D pseudo box. As this is not the primary focus of our innovation, we did not elaborate on it in the main paper. We have updated the details in the supplementary materials.

[To question2, "the illustration in column b is especially egregious"] Our method employs DBSCAN[R2] (a clustering technique) to eliminate noise caused by background points. Given the typically sparse nature of 3D point cloud surfaces, the 3D pseudo-labels might be smaller than actual objects. Consequently, this could lead to pseudo-labels not tightly covering the objects, as discussed in Section 3.1. Additionally, we have replaced Figure 2 with a more illustrative one.

[To question3, posed or unposed"] The probable noise we encounter is shown in Figure 2. The phrase "pose" might lead to mis-understanding, we have rephrased it with "present".

[To question5, "the training process of PRF"] During base training(section 3.5), we train PRF only on base classes. The ground-truth offsets come from the difference between the annotations of the base class and the pseudo 3D labels. In the weakly incremental learning phase, we keep trained PRF frozen.

[To question6, "the input of BCH"]   As mentioned in Section 3.3, BCH serves as a classification head determining whether we retain the 3D pseudo label or not, aiming to address the issue of pseudo-label overlap. BCH takes distinct inputs compared to the PRF module: the input for PRF comprises the scene and coarse 3D pseudo-labels, producing refined pseudo-labels as output. In contrast, BCH takes output from the box encoder and context encoder as input, specifically the box-level features and scene context features.

[To question6, Where does the ground truth "presence" label come from?]   We use the Hungarian algorithm[R3] to match each annotation of the base class with an input pseudo label. The pseudo-labels assigned with ground-truth labels are treated as 1, and the rest are treated as 0. We have updated this part in our manuscript.

Reference

[R1]Bhunia A K, Gajjala V R, Koley S, et al. Doodle it yourself: Class incremental learning by drawing a few sketches[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 2293-2302.

[R2] Ester M, Kriegel H P, Sander J, et al. A density-based algorithm for discovering clusters in large spatial databases with noise[C]//kdd. 1996, 96(34): 226-231.

[R3] Kuhn H W. The Hungarian method for the assignment problem[J]. Naval research logistics quarterly, 1955, 2(1‐2): 83-97.