Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations

We thank the reviewer for their constructive feedback and for recognizing our method’s ability to “generate an object embedding that handles multiple tasks, like visual localization, scene alignment, and scene reconstruction”. We also appreciate their acknowledgment that it “achieves good performance” and their note on its potential to integrate additional pretrained features (e.g., CLIP).

Some feed-forward methods can also reconstruct full 3D even from single RGB, like MIDI.

We thank the reviewer for this suggestion and for pointing us to this recent baseline. We conducted a new experiment comparing Object-X to MIDI on the task of single-image 3D reconstruction using the authors’ released code. Both methods were evaluated in a zero-shot setting on objects extracted from two randomly chosen scenes in the ScanNet dataset, which neither Object-X nor MIDI had seen during training. The quantitative results are summarized below:

As shown in the table above, Object-X significantly outperforms MIDI in geometric fidelity, even in the purely single-image setting (RGB only; same configuration as MIDI). Single-image reconstruction with Object-X from an RGB-D input or reconstruction with the entire pipeline (*) significantly improves geometric accuracy further.

Upon visual inspection, MIDI often appears to mimic the closest shape from the synthetic dataset it was trained on, which rarely coincides with the true geometry of the target object. This aligns with the fact that the original MIDI paper provides no quantitative evaluation on real-world datasets, focusing instead on synthetic data. Combined with our results, this suggests that MIDI may have limited generalization to challenging real-world data such as ScanNet. We will add visualizations to the final manuscript.

Thus, while MIDI remains an excellent and influential contribution, our experiment indicates that Object-X offers a more robust solution for single-image reconstruction in realistic settings. Furthermore, single-image reconstruction is only one of several applications supported by our learned embedding. As demonstrated in the main paper, the core contribution of Object-X is its versatile, multi-modal representation that achieves strong performance across diverse downstream tasks such as visual localization and 3D scene alignment. We will include this new comparison in the final manuscript to further underscore the robustness and generality of our approach.

Adding monocular priors for 2DGS and 3DGS baselines can improve fitting.

We appreciate the reviewer’s suggestion to incorporate monocular priors such as DepthAnything-v2, Stable Normal, or VGGT. While these priors could potentially improve geometric fidelity, they introduce trade-offs: increased storage requirements (e.g., storing floating-point depth maps per image instead of compact descriptors per object) or additional computation time to run a monocular depth network before reconstruction, adding seconds to the pipeline. In contrast, our method achieves state-of-the-art geometric accuracy while using 3–4 orders of magnitude less storage (4.2 KB per object) than RGB images and reconstructing in just milliseconds. That said, we agree that a principled and efficient integration of monocular 3D priors within our framework is a compelling direction for future work, and we will note this in the revised manuscript.

In Full-scene Composition, Object-X Seems to Lose Fine-grained Details.

We thank the reviewer for this observation. As noted in Sec. 4.2 and the Limitations section, full-scene composition is not the primary focus of Object-X. Reduced fine-grained detail in large-scale reconstructions primarily results from the fixed-resolution voxel grid, which under-represents large surfaces (e.g., walls), and missing segmentations of small or thin structures. Despite this, Object-X achieves the highest geometric accuracy while being significantly faster and more storage-efficient than baselines.

Furthermore, applying a lightweight refinement step (Object-X+Opt), within the same time budget as the second-fastest baseline, improves photometric quality to levels comparable to fully optimized 3DGS, while preserving our geometric advantage. This underscores Object-X’s strength as a fast, high-quality initializer for scene-level refinement. We will clarify this trade-off and include additional qualitative examples in the supplementary material.

Clarifications to Reviewer Questions.

• Voxelization algorithm (line 151):
  We scale the object into a unit cube and mark voxels as occupied if they intersect the mesh, utilizing open3d.geometry.VoxelGrid.

• Why not test Object-X on all images (Table 1)?
  We selected 4 (ScanNet) and 12 (3RScan) images per object due to (i) limited additional images observing each object in these datasets, and (ii) fairness in comparison with DepthSplat, which can only process up to 12 images. We will clarify this in the camera-ready version.

• Does “graph2splat” refer to Object-X (Figure 4)?
  Yes, “Graph2Splat” was an earlier name for Object-X. We thank the reviewer for highlighting this and will correct it in the final version.

• Incomplete objects even with multi-view training (Figure 4):
  The incompleteness is a direct consequence of the limited camera coverage in datasets like ScanNet and 3RScan, where parts of some objects are never observed. While optimizing on all available views cannot fully invent unseen geometry, our voxel-grounded representation provides a strong structural prior that encourages plausible shape completion in these sparsely viewed regions. We quantitatively demonstrate this robustness to occlusion in supplementary Table 3, which shows that Object-X can compensate for such partial observations to a degree.

• Scene reconstruction: instance masks and background handling:
  Many methods are available for obtaining 3D object instance segmentation from scene reconstructions, e.g., SceneGraphFusion (CVPR'21), MAP-ADAPT (ECCV'24), Mask3D (ICRA'23), OpenMask3D (NeurIPS'23), and Search3D (RA-L'25). For our experiments, we used the provided annotations on 3RScan and SceneGraphFusion on ScanNet. Regarding background handling, indoor scenes inherently include objects (e.g., walls, furniture) in all pixels of all views. Extending our method to unbounded outdoor settings would require mask pre-filtering, which we consider as future work.

• Distribution differences between ViT and DINOv2 features (line 313):
  The ViT model used for query embedding belongs to the same model family (DINOv2 with additional projection layers) as the one used for generating structured object features, ensuring compatibility.

• Comparison to Trellis (sparse voxel from RGB):
  We implemented a variant inspired by this approach (64³ voxel encoder in supplementary Table 2). This method was memory-intensive (requiring 64³ floating-point numbers compared to our 16³) and generalized poorly to auxiliary tasks due to its sparse representation. Our compressed latent embedding offers significantly lower memory usage and improved task generalizability, while maintaining comparable reconstruction accuracy.

We hope this answer sufficiently addresses the reviewer's comments. Please let us know if any further explanation or clarification would be helpful.

Thanks!

We thank the reviewer for their constructive feedback and for recognizing our framework’s “significant storage reduction” while maintaining strong performance. We also appreciate their acknowledgment of the “encouraging results in multi-task evaluation” and the effectiveness of our method in “object reconstruction, visual localization, and 3D scene alignment”.

Clarity issues.
We thank the reviewer for identifying these issues. We acknowledge that certain notations and explanations in Sections 3.1–3.4 were unclear or inconsistently presented. Specifically, we will:
(i) introduce and clearly explain the object latent embedding in Section 3.1 rather than delaying it to Section 3.2;
(ii) correct the mislabeling of the 3D encoder at Line 158; and
(iii) provide a more explicit and clear description of the components in Section 3.4.

Figure 2 and Pipeline Explanation.
We appreciate the reviewer pointing out the lack of clarity in Figure 2. In the revision, we will redesign Figure 2 to clearly illustrate the relationships among the voxel grid, sparse encoder, structured latent space, and sparse decoder, ensuring all relevant notations (e.g., latent codes, variables) are explicitly included. Unfortunately, due to rebuttal constraints, we cannot present the updated figure here, but it will be included in the final camera-ready version.

Comparison to Previous Works Decoding Gaussians from Latent Space.
We thank the reviewer for this comment. While we would be better able to address this with specific citations, we recognize that some recent works (such as DepthSplat, included as a baseline in Table 1) predict Gaussians from embeddings derived mainly from image features. However, to our knowledge, no previous method employs our approach of compressing object observations into a fixed-size structured latent vector independent of object size, enabling a uniform, highly compact representation that simultaneously supports high-quality reconstruction and diverse downstream tasks. We will clarify this distinction in the revised manuscript.

Description of DepthSplat Integration.
We thank the reviewer for pointing out this omission and apologize for the oversight. In our experiments, DepthSplat is used in its standard pre-trained configuration, applied to the same unmasked scene-level images used by other baselines and our proposed Object-X. Only after reconstruction, object masks are applied to remove background Gaussians, ensuring DepthSplat operates under its intended conditions and aligns with our object-centric evaluation protocol. We will detail this integration in the final manuscript.

Evaluation Lacks Comparison with Stronger State-of-the-Art Methods, such as NoPoSplat.
We respectfully disagree with the reviewer on this point. We benchmark against DepthSplat (CVPR 2025), a recent (published <2 months ago) and relevant state-of-the-art method specifically designed for sparse-view Gaussian reconstruction with known camera poses, making it a robust baseline for our evaluation. NoPoSplat, while an excellent method, addresses the different and more challenging task of jointly estimating camera poses and geometry from sparse input. Our method assumes known poses, making a direct comparison non-trivial without substantial modifications. We will clarify this distinction and explicitly position our contributions in relation to these prior works in the revised manuscript.

Additional single image reconstruction results.
We further compare Object-X to the very recent MIDI method [Huang et al., CVPR 2025] on a subset of the ScanNet dataset. Neither Object-X nor MIDI was trained on ScanNet. While MIDI is specifically designed for single-image 3D reconstruction, Object-X was not explicitly trained for this task but is designed as a versatile, multi-modal embedding for various downstream applications.

The table below reports geometric accuracy for Object-X using only RGB input (directly comparable to MIDI), RGB-D input, and multiple images via the full pipeline (*). Despite its broader scope, Object-X significantly outperforms MIDI even in the RGB-only setting. We will include this result into the final manuscript.

We thank the reviewer for their feedback and for recognizing Object-X as a “compact object latent representation” that decodes to 3D Gaussians with “similar photometric quality and better geometric accuracy”. We also appreciate their acknowledgment of our “voxel-grounded latent representation” and its ability to “augment embeddings with task-specific components”, enabling tasks like visual localization and scene alignment.

Inaccuracy in the Summary.
We thank the reviewer for summarizing our contributions, but would like to correct a misunderstanding: Object-X does not take RGB-D observations as input. As described in Sec. 3.1, our method operates on multi-view RGB images and a coarse geometric representation (mesh or point cloud), which can be obtained from standard multi-view stereo or similar sources. While such geometry may be derived from RGB-D sensors in practice, it is not a requirement of our pipeline. We will clarify this in the paper.

Quality and Texture Detail.
We appreciate the reviewer’s observation. The smoother appearance of our reconstruction stems from the fixed-resolution voxel grid used to ground the latent representation, which trades some high-frequency texture detail for compactness and geometric consistency. This design enables extreme compression (3–4 orders of magnitude reduction in storage) while maintaining strong geometric fidelity (highest F1@0.05 in Tab. 1).

For object reconstruction, this trade-off still results in higher photometric accuracy than 2DGS, 3DGS, and DepthSplat, while significantly improving geometric quality due to the voxelized representation.

For scene reconstruction, as discussed in Sec. 4.2 and the Limitations, reduced photometric quality in the vanilla model arises mainly from (i) fixed-resolution voxels under-representing large surfaces (e.g., walls), and (ii) missing segmentations for small or thin objects. A lightweight refinement step (Object-X+Opt), run for the same time budget as the second-fastest baseline, recovers photometric quality comparable to fully optimized 3DGS (12V) while preserving our geometric advantage and efficiency. This underscores Object-X’s strength as a fast, high-quality initializer for scene-level refinement.

Crucially, Object-X achieves this while producing a single compact embedding usable across diverse downstream tasks (e.g., localization, scene alignment, single-image reconstruction), underscoring the broader utility of our representation. We will clarify this trade-off and add qualitative examples in the supplementary material.

Comparison to TSDF + Marching Cubes.
We thank the reviewer for this suggestion. We note that the ground-truth reconstructions in 3RScan are derived from fused RGB-D frames, whereas Object-X is not an RGB-D reconstruction method but, instead, learns compact object embeddings from multi-view RGB images and coarse geometric priors. While a TSDF + Marching Cubes baseline could provide additional context for pure geometry reconstruction, our goal is to deliver a versatile, multi-modal representation that supports both geometric and photometric reconstruction as well as downstream tasks, rather than performing dense RGB-D fusion. We will clarify this distinction in the final manuscript.

Fairness in the 3DGS/2DGS Comparison.
We thank the reviewer for raising this concern. There is no standardized object-level reconstruction protocol for 3DGS. We chose a sparse-view setting of up to 12 views per object (Sec. 4.1) primarily to ensure a fair comparison with recent learned-prior methods like DepthSplat, which accepts a maximum of 12 views as input. This setting also reflects the natural data sparsity in datasets like 3RScan and ScanNet, where the average number of total views per object is 20 (with ~50% overlap), making our setup a realistic evaluation scenario. Importantly, the 3DGS/2DGS baselines were optimized directly on these views without compression constraints, whereas Object-X encodes them in a compact latent embedding. We will clarify this in the camera ready.

Unfair Comparison to DepthSplat ("evaluated on object-level data with non-overlapping frames").
We believe this misunderstanding stems from our incorrectly formalized sentence at L265, and we apologize for that. The claim that the selected frames are non-overlapping is incorrect: by construction, the selected views for each object do overlap (otherwise, the object would not be visible in all selected frames; ~50% overlap on average). DepthSplat was trained to handle even lower overlaps. This has now been clarified in the paper.

To ensure a fair comparison, we evaluated DepthSplat in the same configuration it was trained for: on unmasked, scene-level images. We only applied the object masks after reconstruction to remove background Gaussians. Thus, DepthSplat was used under its intended operating conditions, making the comparison to Object-X fair. We will revise the text to make this procedure clear.

Storage Computation for Baselines.
We thank the reviewer for raising this important point. For 3DGS/2DGS baselines, we reported storage based on the size of the 12 input images, as their fully optimized Gaussian representations require significantly more storage: on average, 28.41 MB per object, reduced to 7.79 MB (mean, up to 30 MB in some cases) even with aggressive compression (SH=0; leading to reduced accuracy). Object-X encodes objects into a fixed-size latent descriptor of only 4.2 KB, achieving a 3–4 order-of-magnitude storage reduction. Detailed numbers will be included in the supplementary material.

Although advanced Gaussian compression methods exist, our approach uniquely provides:
(i) multi-modal embeddings supporting downstream tasks (localization, alignment),
(ii) extremely fast reconstruction (~150× faster than the next-best baseline), and
(iii) single-image reconstruction capabilities.

Increased Training Time with Downstream Tasks.
We thank the reviewer for this insightful comment. The additional training time introduced by our two-stage training procedure is a one-time cost, as the embedding is trained once and reused zero-shot on other tasks, making this overhead negligible in practice.

Regarding the persistence of benefits when jointly training multiple tasks: as discussed in Sec. 4.3, Object-X jointly trains object reconstruction and localization, achieving strong performance across various tasks, including zero-shot scene alignment and single-image reconstruction. We will clarify this explicitly in the manuscript.

On the justification of training complexity: supplementary Table 1 clearly demonstrates substantial improvements from task-specific training - improving Recall@1, Recall@3, and Recall@5 by 11.8, 10.4, and 6.1 points, respectively. This shows that multi-task training on a shared reconstruction-informed embedding consistently boosts performance, fully justifying the additional complexity.

Confusing technical specifications.
We thank the reviewer for highlighting these points and will revise the manuscript accordingly. Specifically, we will clearly define the voxel occupancy term, restrict the auxiliary encoder’s reconstruction loss to the correct training stage, and resolve all notational inconsistencies (e.g., mislabeling of L). Typos such as "3DSG → 3DGS" in Fig. 5(b) and Table 3 will be corrected. We appreciate the reviewer’s detailed feedback and will incorporate these suggestions to enhance clarity.

Additional single image reconstruction results.
We further compare Object-X to the very recent MIDI method [Huang et al., CVPR 2025] on a subset of the ScanNet dataset. Neither Object-X nor MIDI was trained on ScanNet. While MIDI is specifically designed for single-image 3D reconstruction, Object-X was not explicitly trained for this task as it is designed as a versatile, multi-modal embedding for various downstream applications.

The table below reports geometric accuracy for Object-X using only RGB input (directly comparable to MIDI), RGB-D input, and multiple images via the full pipeline (*). Despite its broader scope, Object-X significantly outperforms MIDI even in the RGB-only setting. We will include this result in the final manuscript.

We thank the reviewer for the comments and are glad that our rebuttal addressed earlier concerns. We also appreciate that the reviewer recognizes our general idea of learning a unified object embedding as a strength.

On-par photometric quality and improved geometric quality.

As the reviewer notes, the improved geometric accuracy stems from the voxel-based representation in Trellis (explicitly cited at L128). This is demonstrated in Table 2 of the supplementary material for photometric quality, with similar trends observed for geometry. Our contribution is to preserve these strengths while reducing the latent representation size by 64$\times$ compared to Trellis. As shown in our ablations, the proposed 3D U-Net is crucial here - naive compression leads to substantial accuracy loss. Moreover, unlike Trellis, whose sparse representation is not readily applicable to downstream tasks, our unified dense embedding supports versatile, multi-task use (as the reviewer also highlighted). We will clarify this in the final manuscript.

"When no depth is provided for TSDF-fusion"

We are slightly unsure what is meant by TSDF-fusion without depth, as TSDF-fusion requires RGB-D input and cannot operate from RGB alone. Regardless, our previous discussion already positions Object-X to Trellis with respect to photometric and geometric quality.

Versatility, zero-shot reusability, and task range.

We agree that exploring a broader range of downstream tasks is valuable. In this work, we focus on five: coarse visual localization (Table 5b), 3D scene alignment (Table 3), fast initialization for scene reconstruction (Table 2), object reconstruction (Table 1), and single-image reconstruction (Table 5a and the one in the rebuttal). While all are geometric in nature, each has its own literature and evaluation protocols, and achieving on-par or superior results to task-specific methods in all of them - zero-shot in some cases - is, in our view, a substantial achievement. For example, we outperform the very recent MIDI (CVPR 2025) significantly by ~14 F1 points on single-image object reconstruction, despite MIDI being designed solely for that task. Given that our learned embeddings capture rich correspondence and reconstruction cues, we see no reason they could not generalize to other applications, such as object classification, which we leave for future work.

Dear Reviewer uwjw,

Could you please review the rebuttal and confirm whether the initial questions or concerns have been addressed? It would also be good to check other reviewers' comments and share your thoughts on related issues. Your participation in this author-reviewer discussion would be greatly appreciated. Thank you very much for your time and effort.

Best,

AC

Follow-up on TSDF-fusion results on RGB-D images

Even though we were slightly unsure about the exact intent of the “TSDF-fusion without depth” request, as TSDF-fusion fundamentally requires depth information to operate, we ran the comparison using available RGB-D object observations (masked to the target object) to ensure fairness against the ground-truth geometry.

As shown below, even with the full set of RGB-D frames, TSDF-fusion underperforms our method in both completion and F1 score. With only 12 views, its completion drops substantially. Our method maintains higher completeness while requiring orders of magnitude less storage. We believe this improved completion is related to our strong single-image reconstruction performance, as our method can infer and reconstruct regions of the scene that are not directly observable in the input views. For efficiency reasons, this evaluation was conducted on a subset of the dataset.

These results confirm that Object-X achieves more complete reconstructions than TSDF-fusion, even when TSDF has access to dense RGB-D input for the entire object. We believe the higher completion scores of Object-X are linked to its ability to plausibly reconstruct parts of the object not directly visible in the input images, a property that also benefits its downstream performance.

We thank the reviewer for their feedback and for recognizing Object-X as a “compact latent representation” that decodes to 3D Gaussians with “similar photometric quality and better geometric accuracy,” and for appreciating its “voxel-grounded design” enabling downstream tasks like localization and scene alignment.

Misunderstanding regarding 3DSG / 3DGS.
We thank the reviewer for raising this point and apologize for the confusion caused by this typographical oversight. In several instances, "3DSG" (3D Scene Graph) was incorrectly used where we intended to write "3DGS" (3D Gaussian Splatting).

To clarify: the core Object-X pipeline does not require 3D scene graphs. Object-X only needs object segmentations as input and learns a unified object embedding that can be decoded into 3DGS for geometry and rendering. Scene graphs (3DSG) are used only as auxiliary inputs for specific downstream tasks such as scene retrieval and localization (see Sec. 4.3), where relational cues between objects are beneficial. Thus, the strong results in object reconstruction (Tab. 1, main paper) and single-image reconstruction (Fig. 4, main paper) are obtained without any 3DSG input. We will fix this typo in the camera-ready version.

Results without 3D Scene Graphs (3DSG).
We provide ablation studies for visual localization in Table 1 of the supplementary material. Specifically, we compare the proposed Object-X without task-specific training (denoted as *), which uses only the proposed U-3DGS embedding, against a variant augmented with additional modalities derived from the 3DSG (i.e., object attributes and relational information). Object-X*, using only the 3DGS embedding (without 3DSG), achieves competitive performance. As expected, incorporating these additional modalities further improves results: Object-X with task-specific training and 3DSG augmentation achieves state-of-the-art Recall@1 and Recall@3, demonstrating that while 3DSG features are not required, they provide complementary benefits when available. We will clarify this in the final manuscript.

Impact of noisy or erroneous scene graphs (3DSG).
As noted above, concerns regarding noisy or erroneous scene graphs apply only to downstream auxiliary tasks (scene retrieval and localization). To evaluate robustness explicitly, we use two distinct scenarios:

• 3RScan, which provides high-quality, manually annotated 3D scene graphs (via SGAligner), serving as a clean, controlled benchmark; and
• ScanNet, which does not have annotated scene graphs, requiring us to automatically predict scene graphs using SceneGraphFusion.

The latter introduces real-world noise, incompleteness, and segmentation artifacts. Object-X demonstrates strong performance on ScanNet despite no retraining (Table 1, main paper, bottom), outperforming baselines in both photometric and geometric metrics. This indicates our learned embeddings are resilient to annotation noise and generalize effectively, even with noisy or incomplete 3DSG inputs.

Full-scene composition quality.
We thank the reviewer for this insightful comment. As explicitly stated in Sec. 4.2 and the Limitations section, full-scene composition is not the primary focus of this work. Instead, our objective is to learn reconstructable, multi-modal object embeddings. The scene composition experiments (Table 2, main paper) demonstrate only the potential viability of Object-X for larger-scale reconstruction with further research.

Reduced photometric quality of the vanilla model primarily arises from two factors:

• the fixed-resolution voxel grid, which under-represents large surfaces (e.g., walls); and
• imperfect object segmentations that omit small or thin structures (e.g., posters, clutter).

Despite these limitations, Object-X achieves the highest geometric accuracy (F1@0.05 = 49.99%), running approximately 150× faster than the second-fastest baseline. Furthermore, after a lightweight refinement step (Object-X + Opt), run for the same time budget as the second-fastest method, Object-X surpasses 3DGS (12V) in photometric quality while preserving its geometric advantage. This underscores Object-X’s strength as a fast, high-quality initializer for scene-level refinement. We will further clarify this in the final version of the manuscript.

Reason for the high geometric accuracy.
We appreciate the reviewer’s observation. The higher geometric accuracy (F1 score) is due to the voxel-grounded nature of our latent representation. Unlike standard 3DGS, which uses an unstructured set of Gaussians, our method first embeds features within a structured voxel grid. This ensures the decoded Gaussians are spatially constrained and surface-aligned, improving geometric consistency and convergence, even from sparse inputs (e.g., single images). In contrast, photometric metrics such as SSIM and PSNR are more sensitive to fine-grained appearance details (e.g., textures), which are partially smoothed due to the fixed voxel resolution. This explains the larger relative improvement in geometric metrics compared to photometric ones. While we cannot include additional visuals in the rebuttal, we will expand this analysis and provide qualitative examples in the camera-ready version.

Additional single image reconstruction results.
We further compare Object-X to the very recent MIDI method [Huang et al., CVPR 2025] on a subset of the ScanNet dataset. Neither Object-X nor MIDI was trained on ScanNet. While MIDI is specifically designed for single-image 3D reconstruction, Object-X was not explicitly trained for this task but is designed as a versatile, multi-modal embedding for various downstream applications.

The table below reports geometric accuracy for Object-X using only RGB input (directly comparable to MIDI), RGB-D input, and multiple images via the full pipeline (*). Despite its broader scope, Object-X significantly outperforms MIDI even in the RGB-only setting. We will include this result into the final manuscript.