Improved Convex Decomposition with Ensembling and Boolean Primitives

$**High computational demand**$: indeed, ensembling involves more compute (see Table 1 for time/memory) but can be worth it when quality matters more than cost. Given that fitting primitives is a very difficult problem, having an avenue to overcome poor initialization/fitting will improve robustness. We also point out that for resource-constrained use-cases where ensembling is not an option, we also make several improvements to the underlying fitting procedure such that any individual network we trained (with or without boolean primitives) compares favorably with prior work (see Tables 1 and 2).

$**Varied datasets**$: We show that our convex decomposition procedure works on general images in Figs. 18-20. We train our model on a subset of LAION (approx. 1.8M varied images) using DepthAnything to obtain a depth map, and reasonable camera calibration assumptions to convert the depth map to a point cloud. Qualitatively, our model abstracts the scene quite well.

$**Explaining prior work**$: It’s challenging to make a paper truly self-contained; we focused the limited space on explaining what’s new and showing results. However, there are only two prior works required to understand what we have done in this work - “CvxNet” and “Convex Decomposition of Indoor Scenes”, and we have made that point in the paper. Further, we have revised the manuscript for clarity and open to improve/explain anything that is confusing.

$**Comparison with Du 2018**$: The key difference is that we fit implicit surfaces to RGB images in an unsupervised way (we do not know in advance the ground truth primitive decomposition). Du 2018 appears to fit primitives to point clouds where the GT CSG shape program is known in advance. Their method doesn’t involve learning, ours does. Their evaluation tries to match the point cloud from the predicted shape program with the ground truth point cloud. We don’t even know in advance if a fixed primitive vocabulary would apply in our approach; Du 2018 knows in advance that the objects can be represented with CSG trees because that’s how they were made. Thus, we expect our objects to have significant error; they would not.

$**Additional CSG grammar**$: Our paper already implements union of primitives (inherited concept from CvxNet/Convex Decomposition) and mesh boolean A(x)-B(x) as described in equation (1) of our paper. It’s conceivable to try other operations like intersection, which can be implemented using our indicators e.g. O(x) = A(x)*B(x), where A(x) and B(x) are implicit surfaces composed of a union of primitives.

$**Evaluation**$: We use the evaluation procedure of “Convex Decomposition of Indoor Scenes 2023” which measure depth, normals, and segmentation accuracy, as well as the error metrics of Kluger 2021. This encompasses both geometric reconstruction quality and object-level alignment of the primitives and the scene. Even though prior works do not use boolean primitives, we still ultimately care about the same things - good geometry and segmentation and therefore our error metrics are reasonable. It is also true that segmentation and scene understanding are more complicated with negative primitives and therefore should be used where we can forego some of the interpretability in exchange for accuracy. Continuing to investigate boolean primitives in the context of scene decomposition is an exciting line of future work - our aim here is to show that they can be fit and improve reconstruction quality in the first place.

$**Ensembling criteria**$: We use best depth accuracy when picking from the ensemble. You raise a good point that picking the best decomposition should take into account both parsimony and depth accuracy. We think such a metric should be fine-tuned based on the downstream use-case; here our goal is to demonstrate how to get better primitives across a wide range of primitive counts (we show 8-40 in the paper).

$**Use of negative primitives**$: We show additional qualitative behavior of the boolean primitives in Figs. 12-17, where they help model bookshelves, chairs, and beds.

$**Other datasets**$: We show our method works on a larger-scale subset of LAION in Figs. 18-20, reporting an AbsRel of 0.130 when using 12 positive primitives with 12 faces each.

$**Ablation on learning rate decay**$: Please see Table 5.

$**Use of bias term**$: Fig. 8 shows that having a small amount of the bias term is generally helpful when fitting boolean primitives, but indeed it’s not a consistent benefit in every case.

$**Theoretical explanation of negative primitives**$: We’re unaware of theoretical methods to model the performance of boolean primitives; however in Fig. 11 we provide an experimental argument showing that fitting boolean parallelepipeds is very difficult, but relaxing the symmetry constraint and Manhattan world loss significantly improves the fitting of boolean primitives.

$**Ensembling selection**$: Indeed our experiments suggest that selecting then refining (pos/pos +neg S->R Table 4) obtains slightly worse AbsRel than 24/32/40 pos only without ensembling. However, the ensembles use 23.3 and 24.5 primitives respectively on average, and we know that more primitives is generally better (up to about 40 primitives). Further, noting that R->S performs substantially better than S->R would indicate that the best start point doesn’t necessarily yield the best end point.

$**Negative primitives reducing fitting accuracy**$: Boolean primitives change the range of geometries we can encode, but they give search problems, as indicated by some scenes showing worse error metrics when replacing positive primitives with negatives. By using boolean primitives, we can encode any scene better if we can find the right fit. But the wide range of geometries we can encode results in variance problems. So in some cases we get better, in other cases we get worse. That’s why we ensemble.

$**Ensembling and fusing predictions**$: What does it mean to fuse all convexes from multiple networks? It was unclear to us how to do that. We agree that our ensembling strategy is simple; we kept it simple because we don’t see how to solve problems created by more complicated strategies. Fusing multiple networks might be much harder than it sounds. We’re looking at a primitive soup of 360 primitives if we consider all 15 members of the ensemble. It’s not clear how to select from this primitive soup. What we have is a selection procedure that convincingly exploits the critical feature of this problem: we can estimate how we’re doing on test data.

$**Comparison with “Robust Shape Fitting 2024”**$: we compare with the conference version of this paper (which shows identical error metrics) in Table 2.

$**Applications of convex decomposition**$: Please see Figs. 18-20.