Jigsaw++: Imagining Complete Shape Priors for Object Reassembly

Thank you for your recognition on this paper. Below we respond to your concerns and hope we can clarify some questions.

• W1: Visualization in size difference in Figure 6(a).

  • Thank you for your suggestion. We have updated Figure 6(a) to better illustrate the challenging size differences our method encounters on PartNet. We hope this visualization helps readers understand the technical challenges in handling large-scale objects within our point cloud-to-image mapping framework.

• W2: How to apply this method to unseen objects?

  • Jigsaw++ demonstrates strong generalization to unseen objects within trained categories, evidenced by our comprehensive results in Table 1 and robustness tests in Table 2 Left. Importantly, all test objects are completely unseen during the generative model training, validating our method's ability to generalize.
  • For novel object categories, performance can be enhanced by expanding the training dataset of the generation module - a straightforward extension and possible future work of our current framework. Our category-agnostic architecture, leveraging image-to-3D mapping, provides a solid foundation for handling diverse object types.

Thank you for your recognition on this paper’s success in involving global information. Below we respond to your concerns and hope we can clarify some questions.

• W1: Reliance on the generation module and the data.

  • This is essentially a problem of data at hand as we discussed in limitation (b)(c). There are indeed geometrical details missing or non-existing parts being added and our method generally relies on the dataset it is trained on, especially the generation part. If the provided shape is completely irregular, then the algorithm would probably give some object-like output.

• W2: Improvement on introduction.

  • Thank you for your feedback on our introduction. We have revised the opening paragraphs to better emphasize that Jigsaw++ aims to enhance existing assembly algorithms by providing predictive complete shape priors, rather than replacing current assembly method. We have also strengthened our contribution statement to highlight this orthogonal relationship with current assembly methods.

• W3: Visualization of point cloud.

  • Our choice of point cloud visualization aligns directly with both the problem formulation and practical considerations in the reassembly domain. While meshes can provide appealing visualizations, the fundamental input representation for assembly algorithms remains point cloud data, as established in the datasets and prior works. Since Jigsaw++ aims to provide point cloud guidance to existing assembly algorithms, we maintain consistency with their input format by using point clouds throughout our pipeline. This choice is not merely presentational - it reflects the actual data representation required for the reassembly task.
  • Moreover, as you noted, while traditional assembly methods output rigid transformations, our goal of providing shape-aware guidance necessitates working directly with point cloud representations. This approach ensures that our method's output can be directly integrated into existing assembly pipelines without additional conversion.

• Q1: function “f” in L248 and visualized point cloud.

  • The function $f$ defined in L248 is a color mapping operator that transforms each 3D point in our point cloud to its corresponding RGB color value. Based on this colorization function, the point clouds are then rasterized to images.
  • In our paper, Figure 8 is a direct visualization of our training and testing data, and how the point cloud looks after applying such a color mapping scheme.

• Q2: Sentences in L251-256 on converting colored images back to point clouds.

  • Our method requires a bidirectional mapping between point clouds and images. While point-to-image conversion uses the color mapping function f, image-to-point conversion reconstructs 3D points from pixel colors and camera poses using the inversion of this function. Multiple views are combined to generate a complete point cloud. The quoted lines discussed such inversion relationships. We have revised L251-256 to provide a clearer explanation of this process.

• Q3: Visualization in Figure 3.

  • The voxel grid visualization in Figure 3 shows the decoded output of the reconstruction latent r, not the latent representation itself. We have updated the figure caption to clarify this.

Thank you for your recognition of our design of the generation model and our strong results. Below we respond to your concerns and hope we can clarify some questions.

• W1: Why not just use the paired data?

  • Our two-stage approach offers crucial advantages over direct paired-data training. The separate generative stage can leverage any 3D data for learning shape priors, not just paired assembly data. While our current evaluation uses dataset-specific training for fair comparison, this architecture enables future incorporation of large-scale 3D datasets without requiring corresponding assembly pairs. This separation of concerns - learning general shape priors first, then adapting to assembly specifics - provides better generalization potential than a monolithic conditional rectified flow.

• W2: Regarding the rectified flow model trained in Sec-4, how effective is the inversion-then-generate process without the "retargeting" step?

  • Our quantitative experiments show that removing the retargeting step significantly degrades performance (CD: 9.6$\times 10^{-3}$, Precision: 45.4, Recall: 43.9), barely improving over baselines. This clearly demonstrates retargeting's necessity in bridging the domain gap between ideal shapes and imperfect assemblies. We provide detailed qualitative comparisons in Appendix Section A and Figure 7 showing how direct inversion-then-generation fails to produce reliable shape priors. These results validate our complete pipeline design.

• Q1: How many shape pairs were used for training during the fine-tuning (or retargeting) of the rectified model? Was the model trained on the assembly results of one model directly applied to others? (e.g., Jigsaw $\to$ SE(3))

  • Our retargeting stage uses 34,072 shape pairs from the Breaking Bad dataset's training set.
  • Importantly, our reported improvements on SE(3) come from a model trained only on Jigsaw outputs, demonstrating strong generalization across different assembly methods (Jigsaw $\to$ SE(3)). These results in Table 1 show that our shape priors effectively enhance assembly performance even when applied to base methods unseen during training. We clarify this training detail in Appendix C.2 in our revised paper.

• Q2: Discussion of existing works: Other model’s results can also be converted to point clouds.

  • Our focus on point cloud generation models is driven by practical considerations of our task's input format. While models like CLAY and X-Cube can indeed generate representations that can be converted to point clouds, the critical challenge lies in converting input point clouds to their required representations. Since our method must process point cloud inputs from assembly algorithms, we prioritize direct point cloud generation approaches. This choice minimizes information loss and computational overhead that would occur in representation conversion, making our pipeline more practical for real-world assembly applications, especially when data is collected by scanning techniques.

• Q3 Q4: Terms for inversion-based methods and discussion on NOCS and X-NOCS.

  • Thank you for your suggestions. We have revised the term “inversion-based methods” in L218-L221 and added a discussion of image-point cloud mapping methods in other domains in Section 4.

Thank you for your recognition that our method works complementary to existing ones and it is adaptable to a wide range of objects. Below we respond to your concerns and hope we can clarify some questions.

• W1: Initial guidance may harm the performance.

  • The partially assembled object is the most informative input for this task, as it encodes valuable spatial relationships from existing assembly algorithms that are absent in unassembled pieces. While using initial unassembled pieces is an interesting direction, our current approach effectively leverages the best available information to generate reliable shape priors that enhance assembly performance. We leave the exploration of using unassembled pieces to future work.

• W2 Q5: Intricate or unconventional objects and real-world samples.

  • Our evaluation encompasses complex categories including “statues” and “toy figures”, which demonstrate significant geometric complexity and unconventional structures. The model handles these challenging cases without any category-specific training, operating in a difficult setting where test objects are completely unseen during training (including the base generation model). This is a notably difficult scenario that goes well beyond “simple” cases.
  • Regarding real-world applications, our method has been rigorously tested on the Breaking Bad dataset which presents more challenging scenarios than existing real-world datasets (e.g., Fantastic Breaks with only two-piece fractures). The dataset features diverse fracture patterns, varying piece counts, complex geometries, and missing parts - conditions that effectively simulate real-world challenges. Per your requests, results on real-world samples from Fantastic Break are included in Appendix E Figure 10, confirming our method’s practical applicability to these cases.
  • We believe that the comprehensive evaluation across these challenging scenarios, supported by quantitative improvements over baselines, demonstrates our method’s broad utility for complex real-world assembly tasks.

• W3: How to ensure the generated results comply with the topic of “fractured assembly”?

  • Our work introduces complete shape priors as a complementary layer of information to assembly methods to help them instead of replace them. The potential positive effect it has towards assembly methods is demonstrated in Table 2 (Right), where incorporating our generated priors improves Jigsaw’s performance by around 50% across all metrics. While current implementation uses closest-point matching to the ground truth shape as proof of concept, our results strongly suggest that developing good shape matching algorithms leveraging these shape priors would significantly advance fracture assembly.

• Q1: Term on whether this is a method designed for assembly algorithms.

  • We revised the introduction and first contribution point of our manuscript. Our paper now clearly positions Jigsaw++ as a complementary tool that provides shape priors for existing assembly methods, not as a replacement.

• Q2: Quantify the shape variance before and after.

  • Our comprehensive metrics (CD, recall, and precision) effectively quantify both necessary and unnecessary shape changes during retargeting. These metrics directly measure alignment with ground truth - any divergence to an incorrect shape would result in significantly worse scores across all metrics compared to the input. The consistent improvements shown in Table 1 across all metrics demonstrate that our method makes meaningful shape adjustments rather than arbitrary changes. For detailed analysis of shape changes, we provide extensive visualizations in Appendix E and metric distributions in Appendix D.1.

• Q3. Would the proposed method generate parts that were not originally from the fractured pieces since it is based on a 3D generation model?

  • Yes, our generative approach may introduce variations from the original pieces - this is an intentional feature that enables our method to imagine complete shapes even with missing or damaged parts, an unavoidable case as we listed in our overview Figure 1. We acknowledge such characteristics in visualization results Figure 11 and limitation discussion Section 6.3.

• Q4 W3: Explain how the proposed approach could be adapted or extended to provide actionable assembly instructions from scanned point clouds.

  • For practical applications like scanned point clouds, our pipeline is straightforward: first apply an existing assembly method (e.g., Jigsaw) to obtain initial piece arrangements, then use Jigsaw++ to generate shape priors. The shape prior would be expected to further guide the assembly method to improve assembly results.

We appreciate the reviewer's concerns, though we respectfully disagree with several points:

• Regarding real-world applicability: The reviewer suggests testing on complex real-world scenarios, yet overlooks that Breaking Bad is currently the most challenging public dataset with realistic multi-piece fractures. The only real-world dataset (Fantastic Breaks) contains merely two-piece samples. We would genuinely welcome suggestions for more complex real-world datasets, as their absence is a fundamental challenge in this field rather than a limitation of our method.

• On shape metrics: We must respectfully point out a misunderstanding here. Since our ground truth is exactly constructed from the original fractured pieces, our metrics (CD, recall, precision) directly measure divergence from the original fractured pieces. This is precisely the quantification of shape distortion the reviewer seeks.

• Regarding assembly uncertainty: The reviewer's concern about generative uncertainty seems to reflect a broader discomfort with generative approaches rather than specific issues with our method. In real-world scenarios, particularly archaeology and restoration, complete certainty is often impossible due to missing pieces and damaged surfaces. This fundamental uncertainty is precisely why generative approaches are valuable - they can suggest plausible completions based on available evidence, just as human experts do. It's worth noting that even expert archaeologists routinely make educated guesses about complete shapes when pieces are missing or during assembly. Our method simply provides a computational approach to this process.

• Regarding usability: The current limitation in applying our shape priors stems from existing matching algorithms (e.g., GeoTransformer) failing to generalize to our scenario - a limitation of those methods, not ours. While developing a new matching algorithm would be valuable, it would make the core problem and contribution of this paper vague (and at least add another 4 pages to this paper, way beyond the conference limit). Our experiments in Table 2 already demonstrate that our shape priors can reduce assembly errors by 50%. This significant improvement over state-of-the-art assembly algorithms validates our approach's effectiveness, regardless of current matching algorithm limitations.

We believe these points demonstrate that our method addresses real practical needs while working within current dataset limitations. The uncertainties the reviewer views as limitations are, in fact, inherent to the problem domain and precisely what our generative approach is designed to handle.

Thank you to the authors for the detailed response. I would like to share my perspective, which may differ from yours, though I fully respect any disagreements:

Regarding the claim that "the only real-world dataset (Fantastic Breaks) contains merely two-piece samples," I believe this is not a sufficient reason to avoid testing on real-world data. It would not be overly difficult to reconstruct real fragments using modern scanning techniques or neural reconstruction approaches using NeuS[1]. While metrics such as CD, recall, and precision are helpful, they are insufficient to fully capture the shape changes introduced by the generative capability. I completely agree with your statement that "developing a new matching algorithm would be valuable," I believe a solution that better couples with the proposed "retargeting" technique would make the paper significantly a strong and complete work for the community, and I look forward to this. Otherwise, I find it difficult to see significant potential for the proposed method to be practically applied to object reassembly tasks, which is the major problem the authors trying to tackle.

[1] NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. NeurIPS 2021. https://arxiv.org/abs/2106.10689