Puzzles: Unbounded Video-Depth Augmentation for Scalable End-to-End 3D Reconstruction

Weakness and Questions

Q1. When using monocular depth maps from MoGe, how is depth consistency ensured between video frames or keyframes? Inconsistent depth could result in poor supervision labels.

• Answer: In the Image-to-Clip experiment, we test two variants (as shown in Fig. 7C): one uses GT depth, and the other uses predicted depth from MoGe to extract geometry from a single image, which is then used to generate clips. In the Clips-to-Clips setting, to ensure depth consistency, we only use GT depth for augmentation and do not include predicted depth.

Q2. I am concerned about whether the proposed method can be applied to videos with lower resolutions, e.g., 512x384. The resized patches may be significantly noisier, which could harm training performance.

• Answer: Yes, applying Puzzles to low-resolution images (e.g., 512×384) may degrade patch quality, which could impact training performance. In our experiments, we use commonly available resolutions such as 1280×720 and 720×1080, which are sufficient to generate high-quality patches and clips. Examples of the resulting outputs can be seen in Fig. 3, demonstrating the visual fidelity preserved at these resolutions.

Q3. Keyframe selection is commonly used in video-based tasks. It is unclear whether the proposed keyframe selection method performs better than those used in existing approaches, e.g., NICE-SLAM.

• Answer: Thank you for the suggestion. We will include a comparison and discussion of SLAM-based keyframe selection methods (e.g., NICE-SLAM) in the main paper. While both approaches involve keyframe selection, their objectives and mechanisms differ fundamentally. In SLAM systems like NICE-SLAM, keyframes are selected to guide online optimization—typically focusing on frames that contribute to pose and geometry refinement through reprojection. In contrast, our method operates on pre-calibrated training data, selecting representative frames based on inter-frame similarity to remove redundancy and ensure viewpoint diversity during data augmentation. Due to these fundamental differences, it is difficult to integrate the two approaches for a fair comparison.

Q4. How efficient is the proposed method for data augmentation? It would be helpful to include an analysis of the time consumption for each step.

• Answer: Using Spann3R as an example, we follow the official training setup and train for 120 epochs on the same dataset. On 8×H100 GPUs, training takes 40 hours with Puzzles and 36 hours without. While integrating Puzzles requires longer training time, the additional 10% is a reasonable and acceptable cost given the performance gains it brings.

Q5. I noticed that the performance of the selected baselines is much worse than that of recent SOTA models such as VGGT. It would be interesting to explore whether the proposed method is effective when applied to more recent SOTA models.

• Answer: VGGT was released in May, around the same time as the NeurIPS deadline (May 15), making it a concurrent work. Due to time and data constraints, we fine-tuned VGGT using the settings from our paper to compare its performance with and without Puzzles augmentation. As shown in the table, on the 7Scenes dataset, Puzzles brings notable improvements—16.8% in ACC and 14.4% in Comp. Like the methods in our paper, Puzzles is compatible with any existing feed-forward 3D reconstruction framework, requiring no changes to the network or training strategy.

Q6. More patch sampling strategies should be discussed and compared in the ablation study. For example, uniformly sampling patches with a fixed overlap ratio or randomly sampling patches, even without overlapping, could be evaluated.

• Answer: We compared different patch sampling strategies in Puzzles. Using a uniform patch sampling method—where each image is evenly sampled with fixed-size patches and overlap—we found it still outperforms the no-Puzzles baseline. However, our proposed random patch sampling strategy achieves the best results. By generating patches of varying sizes, it better simulates diverse camera motions and captures more scene details, leading to improved model performance.

• Q1 A quantitative comparison of camera pose accuracy, with and without using Puzzles.

Due to time constraints, we could not present the complete set of camera-pose accuracy results in the main paper. We will include more results about camera pose estimation in the revision. The tables shown here are from an interim validation and already demonstrate that, in addition to better reconstruction quality, Spann3R with Puzzles yields a significant improvement in camera-pose estimation.

• Q2 The difference in the distribution of camera poses when training with and without Puzzles.

We analyse the difference in camera pose distribution across the dataset used in the main paper.

Table 1. Camera-pose means (μ)

Puzzles follow zero-mean symmetric augmentation leading to no obvious changes in mean.

Table 2. Camera pose standard deviation (σ) before vs. after augmentation

The puzzles augmentation adds obvious variance changes to every axis. The axes in original dataset were already broad (yaw, t_x, t_y) with only small percentage bump. Axes that were narrow (pitch, roll, t_z) balloon dramatically. After augmentation, most rotation axes now sit in the 50 ° – 110 ° range, and translations cluster around 20 m – 35 m. The augmented pose distribution is now more balanced, exposing the model to a wider range of viewpoints. The larger σ values quantify how strongly Puzzles expands this space, leading to better generalisation to rare poses, greater robustness to atypical camera motions, and ultimately higher model performance.

Thanks to the authors for their efforts. I have no further questions. The analysis of camera pose distribution during training and the comparison of camera pose predictions on 7 scenes demonstrate a deep insight into the proposed method. I recommend that the authors add these results to the revision. I will improve my score accordingly.

Weakness

Q1. An ablation study clarify which component are more important

• Answer: We further explore the ablation study of Figure 7.A to illustrate the impact of each component. The experiments show that Clips-to-Clips (C2C) bring the most significant performance gains, while Image-to-Clips (I2C) with augmentation (w/. aug.) provides a smaller but still positive improvement. In scene-level reconstruction, I2C with augmentation helps simulate camera motion, enabling the network to capture local details. In contrast, C2C improves robustness and supports large-scale reconstruction by capturing global context.

Q2. Experiment include 10% data without Puzzles is required.

• Answer: We have supplemented the evaluation with an experiment where only 10% of the training data is used without Puzzles augmentation. The improvement shown in the above table are based on the comparison under the same data size. Compared to the baseline using 10% of the data, applying Puzzles augmentation significantly improves reconstruction quality. Notably, under this limited data regime, the ACC and Comp errors are reduced by 28.2% and 23.7%, respectively. This demonstrates that Puzzles greatly improves data efficiency. While the original setting relies on large-scale data to achieve diversity, Puzzles maximizes the utility of available data by generating more informative and varied training samples.

Questions

Q1. Does the inaccurate pose estimation have an impact of the model performance.

• Answer: We acknowledge this issue, the calibration process is not perfect and can introduce pose inaccuracies. For augmented patches that rely on calibration, we re-render the image using the updated camera intrinsics and extrinsics to align it with the existing pointmap. This step helps correct mismatches between the image and pointmap caused by imperfect calibration, thereby mitigating the impact of inaccurate pose estimation on model performance.

Weakness and Questions

Q1. New views with different intrinsics and extrinsics will introduce artifacts in invisible areas, but may have less effect on the performance of reconstruction tasks.

• Answer: During the method design stage, we observed that camera augmentation can introduce occlusion-induced holes in the projected views, especially in newly visible areas. To address this, we experimented with inpainting to fill these regions. However, our results showed no significant performance improvement, while training time increased by 32.5%. This suggests that such artifacts have limited impact on reconstruction performance, and the cost of mitigating them outweighs the benefits.

Q2. No comparison of this data augmentation to experiments with no data augmentation at all, more traditional data augmentation techniques (cropping, jittering, image rotation, etc.), and this data augmentation to provide a more comprehensive understanding of how important the perspective projection augmentation is for the reconstruction task.

• Answer: We compared the performance of Spann3R on the 7Scenes dataset under three settings: (1) no augmentation, (2) traditional augmentation (cropping + jittering + flipping), and (3) our proposed Puzzles. In the official implementation, traditional augmentation includes cropping, jittering, and flipping, which enhance appearance diversity at the image level. In contrast, Puzzles augments data from a spatial perspective by simulating camera motion and introducing diverse viewpoints. Our results show that Puzzles leads to more effective generalization, outperforming both the no-augmentation and traditional augmentation settings.

Q3.It is unclear whether the model performance comparison with and without data augmentation uses the same number of training epochs, and the impact of data augmentation on training efficiency and convergence rate.

• Answer: Using Spann3R as an example, we follow the official training setup and train for 120 epochs on the same dataset. On 8×H100 GPUs, training takes 40 hours with Puzzles and 36 hours without. We observe slower convergence when using Puzzles, likely due to increased data diversity and complexity it introduces. While longer training could improve performance, we use the same settings for fair comparison.

Weakness:

Q1. Lack of 1) background motivation, 2) clear target function and 3) detailed data processing.

• Answer: 1) Data-driven feed-forward 3D reconstruction methods, such as VGGT, have shown great promise, with data quantity and quality of training data being key to their success. Existing feed-forward methods primarily rely on 2D data augmentation techniques such as cropping, clipping, and jittering, without incorporating any 3D-aware augmentation. In contrast, our method enhances viewpoint diversity by explicitly controlling camera motion, leading to a more diverse and balanced data distribution in 3D space. 2-3) The task involves taking multiple images (extracted from a video) as input, without requiring any camera parameters, and outputs a pointmap (point cloud) for each image. All pointmaps are expressed in the coordinate frame of the first image. The training data consists of video clips along with their corresponding pointmap.

Q2. Hard to tell how much the specific design choices are optimal, which lacks clear definition and quantitative analysis.

• Answer: Directly evaluating the effectiveness of data augmentation is inherently challenging. Instead, we provide both an analysis of the augmented data distribution and its impact on feature representations. On the ScanNet++ dataset, we observe that the original camera pose distribution is dense along the Yaw and Pitch axes but sparse along the Roll axis, due to the nature of indoor scanning trajectories. After applying Puzzles, the distribution becomes more balanced across all three axes. At the feature level, we extract representations using Spann3R models trained with and without Puzzles. Features obtained with Puzzles exhibit a more uniform distribution, indicating better coverage of the representation space. While we are unable to include figures in the rebuttal due to space constraints, we will provide detailed visualizations and expanded discussion in the main paper.

Q3. Design choices are heuristic, need a general way.

• Answer: While our method involves heuristic components, similar to many existing data augmentation techniques, it is designed to be general and broadly applicable. Puzzles functions as a plug-and-play module that can be integrated with any feed-forward 3D reconstruction model, whether it operates on image pair or sequences. It requires no modification to the network architecture and can be seamlessly incorporated at the data loading stage to enhance viewpoint diversity. Furthermore, our method aligns well with current training paradigms. Many reconstruction models adopt local-to-global training strategies, where the early warm-up stages involve high data overlap, and later stages reduce overlap to promote broader scene coverage. Puzzles naturally supports this approach: its Image-to-Clips module allows control over patch overlap, allowing flexible adaptation of augmentation strength across training phases.

Q4. Lack of experimentation details including training scheme, training data, on the fly or pre-processing, training time, iteration and convergence.

• Answer: Using Spann3R as an example, we follow the official training setup with the same 14M training samples (ScanNet-v2, ARKitScenes, Habitat subset, and BlendedMVS). The only change is the application of Puzzles augmentation. Puzzles is a plug-and-play module applied on-the-fly through the dataloader, without requiring any pre-processing or modification of the training pipeline. Training was conducted on 8×H100 GPUs for 120 epochs. The total training time was approximately 40 hours with Puzzles, compared to 36 hours without. We observed slightly slower convergence when using Puzzles, which is expected due to the increased diversity and complexity of the training data. While longer training could yield further improvements, we use the same configuration across experiments to ensure fair comparison.

Questions

Q1. More details about augmentation

• Answer: Our augmentation method is designed to be easily integrated into any feed-forward 3D reconstruction training pipeline. For each training batch, augmentation is applied at the video clip level. Given a clip with N images and their corresponding point clouds, we first select key representative frames using an overlap-based clip-to-clip matching matrix. Then, we apply image-to-clip augmentation to synthesize new training samples. The augmented data retains the same format as the original—each clip contains of a sequence of images along with their corresponding pointmap. This consistency ensures compatibility with existing architectures and training workflows.

2. Image-to-Clips

Q2.1. How are the patches actually selected?

• Answer: To generate N patches from a single image, the first patch is generated randomly. Each subsequent patch is then created to ensure it overlaps with at least one of the previously generated patches, based on IoU calculations. This overlap constraint helps maintain spatial continuity and supports consistency across views. The patch size is determined dynamically, based on the image resolution and the desired number of patches per image.

Q2.2. In patch calibration: a. Why do you allow changing either intrinsics or extrinsics, but not both at the same time? b. When using fixed-intrinsics-varying-extrinsics, Doesn't it incur only change in translation? (You say it doesn't create realistic rotation dynamics) c. Why is RanSAC needed in PnP? What are the outliers?

• Answer:
• a: Varying both intrinsics and extrinsics simultaneously is theoretically possible, but it causes unnecessary computational overhead. Given that our training data consists only of images and their corresponding pointmap—without access to ground-truth camera parameters—we prioritize simplicity and robustness. A lightweight calibration strategy suffices to generate diverse and effective image–pointmap pairs.
• b: Without augmentation, generating patches only simulates camera translation. As shown in Fig. 2C, all cameras views remain largely forward-facing, limiting rotational diversity. In contrast, our augmentation introduces varying extrinsics, which results in more realistic and diverse camera orientation, as illustrated in Fig. 2D.
• c: The training data includes real-world samples collected via sensors and manually calibrated, which inevitably introduces noise and outliers. Using RANSAC PnP helps mitigate the effect of these errors. It helps filter out erroneous correspondences between 2D image features and 3D points that could otherwise degrade pose estimation.

Q2.3. No enforce overlap between consecutive frames.

• Answer: Yes, in our augmentation strategy, each newly generated patch is required to overlap with at least one of the previously generated patches, but not necessarily with consecutive frames. This design choice accommodates scenarios with rapid camera motion, where adjacent frames may have little or not overlap.

Q2.4. The impact of filling occluded regions with white colour.

• Answer: We also observed this issue. We applied LAMA [1] to inpaint occluded regions and excluded the inpainted areas from supervision. However, our experiments conducted on 8×H100 GPUs show that inpainting does not provide a significant performance improvement, while increasing training time by 32.5%. Additionally, existing inpainting methods tend to introduce artifacts when filling large missing regions, contaminating the valid data. Therefore, we decided to discard this approach.

Reference:[1] Resolution-robust Large Mask Inpainting with Fourier Convolutions

Q3. Clips-to-Clips. Render a new sequence on point cloud during training.

• Answer: Pre-processing Hypersim data using this method is time-consuming—it takes about 3 seconds per clip and around 9 hours for the entire dataset. Therefore, re-rendering scenes on the fly during training is currently not feasible. That said, we agree it is a promising direction, especially for large-scale scene reconstruction (as mentioned in the supplementary), where stitching multiple scenes and generating expansive training data could offer significant benefits.

Q4. Table 1: Are your results using clip-to-clip or image-to-clip? (And same question regarding Figure 7 B and C).

• Answer: Thanks for pointing out. The results in Table 1 and Figure 7 B are based on the clip-to-clip setting, while Figure 7C corresponds to the Image-to-Clip setting.