Rig3R: Rig-Aware Conditioning and Discovery for 3D Reconstruction

We thank the reviewer for their encouraging and thoughtful feedback. We appreciate the recognition of our contributions and the insightful questions, which help clarify key aspects of the work. Below we address each point in detail and will revise the final version accordingly.

1. Reconstruction from Non-Overlapping Image Pairs and Input Frame Constraints

Yes, Rig3R can indeed reconstruct non overlapping image pairs, particularly in the case when it is calibrated. The rig metadata, such as rig extrinsics and the image timestamps, provide strong constraints for the reconstruction that allow the model to pose the images even with no visual correspondences. We find that most rigs often have little overlap in their field of view, but Rig3R still successfully reconstructs their captured scene. In terms of input constraints, Rig3R is trained with frame ID interpolation similar to Fast3R, enabling generalization to variable-length sequences at inference. While most evaluations use 24-frame inputs for consistency, we have successfully run inference on 50+ frame sequences using a single A100 GPU. The theoretical limit is 100 frames, as this was the upper bound for our frame ID interpolation. We will be sure to include these findings in the final version for clarification.

2. GPU Memory Consumption at Inference

The model itself takes up 10GB of VRAM, allowing it to fit in many standard GPUs. For every input frame the model processes at 512x512 resolution, an additional 0.85GB of VRAM is required. So, a 24 frame input requires roughly 30.5GB of input, and an A100 GPU can support roughly 82 frames per forward pass. This memory footprint is comparable to other learned feedforward reconstruction methods. We will add this resource usage breakdown to the final paper to improve reproducibility and clarify hardware requirements.

3. Comparison with VGGT

We experimented and found that for in-distribution data, such as Waymo, $Rig3R_{unstr}$ and VGGT perform similarly, and $Rig3R_{calib}$ with the rig metadata largely outperforms VGGT on both pose estimation and pointmap metrics. On out-of-distribution data such as WayveScenes 101, we find that VGGT outperform the base $Rig3R_{unstr}$, but with the rig metadata, $Rig3R_{calib}$ still strongly outperforms VGGT. We find this to be strong support for the central claim, that embedding rig metadata provides a strong constraint for reconstruction and pose estimation and allows for better generalization.

In terms of efficiency, Rig3R is a single-pass feedforward model with comparable inference time and memory footprint to VGGT. Once again, we thank the reviewer for their thoughtful and balanced review. Your comments helped us clarify key aspects of our method and improve the presentation. We believe the planned revisions will significantly strengthen the final version.

We thank the reviewer for their encouraging and thoughtful feedback. We appreciate the recognition of our contributions and the insightful questions, which help clarify key aspects of the work. Below we address each point in detail and will revise the final version accordingly.

1. Support for Variable Numbers of Frames

The reviewer is correct, our model can support a variable number of frames. During training time 24 frames was a fixed parameter, and we used sample frame IDs from an ID pool similar to Fast3R to generalize to more frames. While the main experiments use a fixed 24-frame input for consistency and comparability across baselines, we have validated the model's generalization to different sequence lengths.

Specifically, we tested sequences from 1 frame (monocular depth) up to 82 frames, constrained only by GPU memory. The model retains stable performance across this range. The theoretical maximum is currently 100 frames, based on the size of the learned embedding pool.

That said, longer sequences can pose challenges, particularly for aligning distant views to the first frame, due to accumulated pose drift. This is a known issue in transformer-based sequence modeling, but not the focus of our current work. We will clarify this behavior and include supporting empirical results in the revision.

2. COLMAP Baseline Setup and Performance

The COLMAP ground truth used in WayveScenes101 was generated using full, dense sequences, which benefits from abundant overlap and context. In contrast, our evaluations focus on more challenging settings involving strided sequences with large baseline shifts, where keypoint overlap is more sparse. COLMAP suffers in these cases due to limited tracking opportunities, though it benefits from being able to jointly optimize over many frames. To stay consistent with learned baselines and our training, we process in 24 frame chunks. Feedforward models are limited in that they treat each chunk as an independent sample. To avoid artificially handicapping COLMAP, we allow it to process and optimize the full strided sequence as a whole rather than independent chunks. We will clarify this setup more explicitly in the final version.

3. Clarification on Rig3R_Unstr Performance

Thank you for catching this discrepancy. We agree that $Rig3R_{Unstr}$ does not outperform all feedforward baselines and will revise our wording to reflect this. We will revise this section to reflect that $Rig3R_{unstr}$ performs comparably to other feedforward methods, but does not outperform them across all metrics. It exhibits a clear drop in performance relative to $Rig3R_{Calib}$, highlighting the importance of rig priors when available. Nevertheless, it remains a viable zero-shot option when rig metadata is missing, and this flexibility is what we intended to show

4. Chamfer Distance and Pose Accuracy

Our intention was not to frame lower Chamfer as a direct advantage. Rather, we observed that Fast3R achieves higher Chamfer despite poor pose accuracy, suggesting that using intermediate pointmaps to infer pose (e.g., via PnP) may not guarantee consistent geometry. The key advantage is directly optimizing for pose via regression on our raymap output representation without relying on pointmap performance, yielding more consistent multi-view reasoning.

5. Metadata Dropout and Generalization

The reviewer is correct in noting that dropout during training is a key factor in enabling generalization both to settings with missing rig metadata and to varying rig configurations. While we do not tune this as a hyperparameter, we did experiment with different dropout levels to assess its impact. As expected, training with no dropout leads to poor performance when metadata is absent, while full dropout reduces the model to Rig3R_unstr, removing the benefit of rig constraints. We found that at scale, training with intermediate dropout naturally exposes the model to a wide spectrum of metadata availability, making the specific dropout rate relatively insensitive. We use 50% as a standard, untuned setting that balances exposure to both structured and unstructured regimes. We will clarify and give more context to this in the final version.

6. Loss Function and Rig-Relative Consistency

We did not include an explicit rig-relative consistency loss. However, the rig-relative raymaps are naturally aligned for cameras on the same rig and encouraged to remain consistent via shared supervision targets and metadata when available. We agree this is an interesting direction and will note it as future work.

7. Missing Frustums in Fig. 4

The missing frustums are a product of taking 24 frame outputs with variable number of rig cameras. To remain consistent with the main table’s setup, our qualitative outputs also use 24-frame samples. In configurations with more cameras (e.g., 7-camera rigs), this corresponds to fewer timesteps, with the final timestep including only a subset of cameras (e.g., 3 out of 7). This is an intentional choice for the figure and highlights Rig3R’s ability to handle partial rigs and missing views, illustrating our model’s flexibility to operate even when the rig is incomplete. Thank you for this note, we will be sure to clarify this in the figure caption!

Once again, we thank the reviewer for their thoughtful and balanced review. Your comments helped us clarify key aspects of our method and improve the presentation. We believe the planned revisions will significantly strengthen the final version.

We thank the reviewer for their encouraging and thoughtful feedback. We appreciate the recognition of our contributions and the insightful questions, which help clarify key aspects of the work. Below we address each point in detail and will revise the final version accordingly.

1. Training Cost and Accessibility

This is an important concern, and we appreciate the opportunity to clarify. We trained the full-scale Rig3R model using 128 H100 GPUs to maximize training diversity and batch size, which we found to be beneficial for generalization to unstructured and unseen rig configurations (e.g., WayveScenes101). However, the model architecture itself is lightweight, requiring approximately 10 GB of memory at inference, and can run efficiently on standard single-GPU setups. To address accessibility, we also conducted ablations at reduced scale (32 H100 GPUs over 2 days), which is already smaller than comparable works such as Fast3R. These smaller models still demonstrate strong performance, particularly when rig metadata is provided, as shown in Tables 3 and 4.

2. Ablation of Rig Metadata Modalities

We thank the reviewer for highlighting this, we agree that understanding the contribution of each modality is critical. Table 3 reports performance when only a single metadata input (camera ID, timestamp, or rig pose) is provided on our final model, showing isolated effects of the metadata on the model performance. We observe that rig-relative pose information provides the most direct geometric constraint, and is especially helpful for inferring consistent camera poses. However, camera IDs and temporal indices still yield performance gains, particularly when used together, as they capture useful ordering and grouping priors. We will make this analysis clearer in the final version and highlight any observed redundancy or complementarity between signals. Table 4 compares structured, unstructured, and downscaled training settings, providing insight into how the full metadata combination performs. Due to the computationally expensive training of the full-scale model, these ablations were performed on fewer GPU’s with a smaller batch size.

3. Qualitative Rig Discovery Evaluation and Failure Cases

We agree that qualitative insights would greatly aid interpretability. We will aim to include more qualitative results on rig discovery in the final version of the paper. In the supplementary material, we currently include visualizations of 3D reconstructions under different rig metadata conditions and color-coded raymaps and camera centers, illustrating discovered rig configurations. We will expand this further to include 3D layouts of inferred rig geometry, showing consistency across scenes and highlighting discovery dynamics. As for failure modes, the most common include unusual or novel rig layouts that diverge significantly from training distributions where the rig calibration is not present, and also highly dynamic scenes. We will document representative failure cases in the revision and supplement.

4. Generalization Beyond Autonomous Driving We agree that exploring non-driving scenarios involving multi-camera rigs, such as indoor robot rigs, AR/VR capture systems, would be a valuable extension of this work and an exciting direction for future research. Our current focus is on the challenges specific to real-world rig distributions, where cameras often have minimal field-of-view overlap, making it especially important to leverage rig constraints. To highlight these effects, we evaluate on publicly available rigged datasets, which are predominantly from the autonomous driving domain due to their scale and accessibility. We will look to explore additional domains and non-driving datasets in future work.

Once again, we thank the reviewer for their thoughtful and balanced review. Your comments helped us clarify key aspects of our method and improve the presentation. We believe the planned revisions will significantly strengthen the final version.

We thank the reviewer for their encouraging and thoughtful feedback. We appreciate the recognition of our contributions and the insightful questions, which help clarify key aspects of the work. Below we address each point in detail and will revise the final version accordingly.

1. Clarification of Model Outputs and Prediction Heads

The model outputs include: (1) dense pointmaps in the coordinate frame of the reference image, (2) dense pose raymaps in the same frame, and (3) rig-centric raymaps in the coordinate frame of the reference camera.

The choice of prediction head is based on the type of information rather than output density. Pointmaps require fine-grained, pixel-level, spatially localized features and benefit from the multi-scale structure of the DPT head. Raymaps, on the other hand, encode global pose information per frame and do not require such detail. We therefore use lightweight MLP heads to avoid unnecessary complexity. We will clarify this design choice in the paper.

2. Differences Between $Rig3R_{unstr}$ and Fast3R

$Rig3R_{unstr}$ operates without rig metadata at inference and is functionally similar to Fast3R in that sense. However, our training includes metadata dropout and rig-aware output heads, which encourages the model to learn more consistent internal representations. This is particularly true for rig configurations seen in training, leading to higher performance of $Rig3R_{unstr}$ on Waymo. This implicit structure could help improve spatial coherence, potentially explaining the stronger mAA and RTA scores observed for WayveScenes101.

3. Rotation vs Translation Metrics

The reviewer is correct in noting that Rig3R performs better on translation. This is indeed a function on our training mix, which largely includes time-strided driving data. This distribution encourages the model to develop a strong understanding of vehicle dynamics and temporal structure, particularly when coupled with rig poses and timestamps during training. As a result, the model may implicitly focus on spatial consistency in translation. We would like to study this behavior further in future directions.

4. Evaluation Beyond Driving Datasets

We agree with the reviewer that it would be valuable to explore non-driving scenarios with multi-camera rigs, such as multi-view drones or robotics platforms, and we consider this a valuable direction in future work. Our paper and model focus on the challenges posed by the rig distribution, where cameras often have minimal field-of-view overlap, and how this makes leveraging known rig constraints particularly beneficial. To demonstrate these effects, we evaluate on public, real-world, rigged datasets, most of which are commonly driving datasets.

5. Temporal Coverage and Input Setup

Our main evaluations use 24-frame input sequences, corresponding to 5 distinct timesteps (with a 2-second stride) across 5 cameras, covering approximately 10 seconds. Section 4.3 explores different camera counts and the associated trade-offs in the number of timesteps. While we fix the input to 24 frames for consistency in evaluation, the model supports and has been tested with more or fewer frames. However, this flexibility is not a primary focus of the paper.

6. Using Both Camera and Time IDs (Uncalibrated Rig Setting):

We agree this is a meaningful intermediate case, where the presence of a rig and timestamps is known but calibration is unavailable. We expect its performance to fall between the fully unstructured and calibrated variants, and plan to include experiments with this configuration in the final version.

Once again, we thank the reviewer for their thoughtful and balanced review. Your comments helped us clarify key aspects of our method and improve the presentation. We believe the planned revisions will significantly strengthen the final version.