Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splats

1. Comparison with recent 3DGS/NeRF variants, such as Zip-NeRF, Scaffold-GS, and Mip-Splatting.

Ans: We include a comparison table with Scaffold-GS and Mip-Splatting below, omitting Zip-NeRF. Although Zip-NeRF achieves higher rendering quality, it requires dozens of hours to optimize a single scene, which is an order of magnitude slower than GS methods that typically take around 10+ minutes. In particular, under the same input setting, Mip-Splatting achieves similar performance to 3D GS while Scaffold-GS leads to superior quality.

It is important to note that the contributions of these works are orthogonal to ours. Our work focuses on large-scale feed-forward GS, addressing challenges in scalability and efficiency. We leverage the original 3D GS representation in our model. In contrast, Mip-Splatting and Scaffold-GS focus on improving the 3D GS representations, instead of developing feed-forward solutions. Our approach could potentially be extended to incorporate these advanced representations, which we leave as a direction for future research.

Rendering quality on DL3DV-140 with 32-view input

(The number for Long-LRM slightly improves over the number reported in the original submission because, during the development of Long-LRM, only the first 7K scenes of the DL3DV10K dataset were available for training. The updated numbers, presented here, come from a version of Long-LRM trained on the full 10K scenes.)

2. Video results show floaters, suggesting the need for further regularization.

Ans: We address the challenging task of wide-coverage scene reconstruction from 32 input images, a problem made difficult by the relatively limited view distribution in large scene spaces. Usually, standard per-scene optimization-based NeRF and 3D GS methods would require hundreds of input views to reconstruct such a large scene space. In our setting of 32 input views, even the standard per-scene optimization-based 3D GS can result in floater artifacts. However, as demonstrated in our qualitative comparisons with 3D GS in the paper, Long-LRM shows significant improvements in reducing floaters. This improvement can be attributed to two key factors. First, as a feed-forward method, Long-LRM leverages prior knowledge distilled from a large training dataset, helping to avoid floaters in unseen views. Second, we have already adopted regularization terms like the opacity loss and the soft depth supervision, which are effective in mitigating floater artifacts.

Overall, wide-coverage feed-forward scene reconstruction remains a relatively new and underexplored problem. Our work makes a step by scaling feed-forward models to support 32 input images to address this problem. We look forward to future advancements that further scale feed-forward models to handle even more input views—potentially several hundreds—which could further reduce floaters and improve reconstruction quality.

7. Whether Long-LRM can seamlessly concatenate predicted Gaussians from multiple sets of images.

Ans: Currently, Long-LRM is not trained to seamlessly fuse Gaussians predicted from multiple sets of images. Optimization-based methods like CityGaussian achieve this by first optimizing a coarse set of Gaussians for the entire scene, which serves as a prior. The scene is then divided into smaller blocks and optimized independently, which can be directly concatenated thanks to the reference provided by the coarse Gaussians.

In contrast, Long-LRM, as a feed-forward method, predicts Gaussians from a set of images in a single pass without any additional knowledge of concurrently or previously generated Gaussians from other sets. Seamlessly concatenating all sets of Gaussians may require additional post-processing steps, such as filtering and merging techniques. An exciting future direction would be to further scale feed-forward GS models to handle hundreds of input images directly.

3. Technical novelty.

Ans: We emphasize that our work targets a new and under-explored task: achieving feed-forward, wide-coverage 3D scene reconstruction from a large number of input images. Prior to our work, large scene-level NeRF or 3DGS reconstructions could only be achieved by per-scene optimization methods. Recent feed-forward methods, such as GS-LRM, Gamba, and MVGamba, have made progress but focus on object-level reconstruction or sparse input views, leaving wide-coverage scene reconstruction largely unexplored. Our work is the first to demonstrate that feed-forward models can be effectively designed to handle large-scale scene reconstruction tasks, which we believe provides valuable insights to the community.

A key aspect of our contribution is addressing the challenge of long-sequence inputs with up to ~250K tokens—a scale more than ten times larger than existing transformer- or Mamba-based 3DGS models. This significantly longer sequence introduces unique challenges in maintaining high reconstruction quality while managing computational efficiency.

To tackle these challenges, we propose a novel hybrid architecture that combines transformers and Mamba2 blocks, achieving a balance between quality and efficiency. While similar hybrid architectures have been applied to other vision tasks, we are the first to adopt this hybrid approach for 3D reconstruction. In addition, we introduce token merging and Gaussian pruning techniques to further reduce computational complexity, enabling our method to handle significantly longer sequences compared to prior methods.

We also emphasize that training at this large input scale is inherently more challenging and unstable than training on shorter sequences. To address this, we designed a multi-resolution training strategy with combined loss functions, enabling stable training at large scales. To the best of our knowledge, this is the first feed-forward 3D reconstruction solution demonstrated at this large input scale with up to 32 images at a resolution of 960x540. Furthermore, we conducted a comprehensive study and analysis of various design choices at the same large input scale; we hope this provides helpful insights into the trade-offs and performance of different techniques for long-sequence reconstruction tasks.

We hope our work inspires the community to develop new technologies to address the challenges of the under-explored task of feed-forward wide-coverage 3D scene reconstruction.

4. Sparse-view NVS comparison with generalizable 3DGS methods like MVSplat and pixelSplat.

Ans: As shown in the table below, Long-LRM outperforms all previous cost volume-based feed-forward GS methods under sparse settings and performs on par with GS-LRM without the token merging step. The token merging introduces a trade-off between scalability and model capacity. While it slightly reduces quality in this sparse-view setting, it enables Long-LRM to handle higher resolutions and longer sequences, addressing scalability challenges effectively.

Rendering quality on RealEstate10k with 2-view input

5. Surface reconstruction results should be included.

Ans: We have updated our project page to include a video interleaving the color reconstruction and the geometry reconstruction. Please find this video at the top of the page (https://longgggglrm.github.io/).

6. Regularization for aliasing effects.

Ans: Our work focuses on feed-forward wide-coverage 3D GS scene reconstruction, and addressing anti-aliasing is orthogonal to the scope of this study. It is worth noting that none of the existing feed-forward 3D GS methods (e.g. pixelSplat, MVSplat, GS-LRM) address the aliasing effect either. Extending feed-forward models to handle aliasing is indeed an interesting and valuable future direction, and we look forward to seeing advancements in this area.

Long-LRM is not designed for an online reconstruction setting. It does not anticipate users sending images one by one to acquire reconstructions on a per-frame basis. Our focus has always been on the offline setting, where a user provides all input images at once, and the model outputs a global reconstruction, same as the optimization-based methods. We emphasize that our method, with the same experiment setting, achieves a 700x speed-up compared to all optimization-based methods, delivering competitive quality with only a 0.8dB drop compared to Scaffold-GS, while outperforming the original 3D GS and Mip-Splatting. While the method is not online, we argue that, in the context of offline setting, this is a substantial improvement over a 10-min wait required by the optimization-based techniques.

In addition, we respectfully disagree with the reviewer's claim that rendering quality should be the sole standard for publishing an academic paper. In the field of 3D vision and scene reconstruction, speed and efficiency are widely recognized as critical contributions. Many highly impactful works, such as SNeRG (ICCV 2021) and MobileNeRF (CVPR 2023), were published with a primary focus on reconstruction or rendering efficiency improvements, despite not achieving state-of-the-art rendering quality. Both papers were highly impactful and widely recognized for their efficiency contributions, with SNeRG accepted as an Oral in ICCV 2021 and MobileNeRF accepted as an award candidate in CVPR 2023.

Similarly, our work demonstrates a clear advantage in reconstruction efficiency, achieving a significant speed-up of two orders of magnitude with minimal performance loss relative to state-of-the-art methods. We hope the reviewer will follow the community standards in evaluating our work and recognize the importance of improvements in efficiency. Unbiased reviewing should focus on overall relative advancements rather than requiring perfection across all metrics for a research paper.

Furthermore, as the reviewer noted, feed-forward approaches were traditionally only demonstrated in sparse-view settings. Our work is the first feed-forward model to show competitive quality to optimization-based methods in a dense-view wide-reconstruction scenario. This extends the scope of what feed-forward methods can achieve. We view this as an important first step and look forward to future works that build on this direction to achieve even better quality, potentially reaching state-of-the-art rendering quality.

Finally, we would like to mention that for Scaffold-GS, its specialized GS representation stores features and offsets instead of spherical harmonics coefficients, making direct initialization with Long-LRM's predicted Gaussians incompatible, while initializing with only point positions would lead to suboptimal results. Therefore, we consider Scaffold-GS's contribution orthogonal to our study that is based on the original 3D GS representation. Exploring advanced feed forward models to directly predict Scaffold-GS-based features and offset parameters is an interesting direction for future research.

3. Discussion on prior Mamba-based 3D reconstruction methods, such as Hamba, Gamba, and MVGamba.

Ans: We thank the reviewer for pointing out the missing citations. We will include and discuss these relevant works—Hamba, Gamba, and MVGamba—in a revision. In particular, Hamba is a Mamba-based method specifically designed for hand reconstruction, while Gamba and MVGamba utilize Mamba for feed-forward 3D GS modeling, which aligns more closely with our task. However, these methods adopt pure Mamba architectures and focus on object-level sparse-view reconstruction, typically using only 1–4 input images and processing around 10K patch tokens. In contrast, Long-LRM is designed for scene-level reconstruction from a significantly larger number of input images (up to 32) with higher resolutions, presenting key new challenges in processing long-sequence input tokens. Instead of relying on a pure Mamba architecture, Long-LRM employs a hybrid architecture that combines transformer and Mamba2 blocks, enabling efficient processing of up to 250K tokens. This scale is far beyond what previous Mamba-based methods handle. Our contributions specifically target the scalability and the unique requirements of large scene-level reconstructions with long-sequence inputs, which go beyond the scope of methods like Gamba, MVGamba, and Hamba.

4. Technical contribution.

Ans: We emphasize that our work targets a new and under-explored task: achieving feed-forward, wide-coverage 3D scene reconstruction from a large number of input images. Prior to our work, large scene-level NeRF or 3DGS reconstructions could only be achieved by per-scene optimization methods. Recent feed-forward methods, such as GS-LRM, Gamba, and MVGamba, have made progress but focus on object-level reconstruction or sparse input views, leaving wide-coverage scene reconstruction largely unexplored. Our work is the first to demonstrate that feed-forward models can be effectively designed to handle large-scale scene reconstruction tasks, which we believe provides valuable insights to the community.

A key aspect of our contribution is addressing the challenge of long-sequence inputs with up to ~250K tokens—a scale more than ten times larger than existing transformer- or Mamba-based 3DGS models. This significantly longer sequence introduces unique challenges in maintaining high reconstruction quality while managing computational efficiency.

To tackle these challenges, we propose a novel hybrid architecture that combines transformers and Mamba2 blocks, achieving a balance between quality and efficiency. While similar hybrid architectures have been applied to other vision tasks, we are the first to adopt this hybrid approach for 3D reconstruction. In addition, we introduce token merging and Gaussian pruning techniques to further reduce computational complexity, enabling our method to handle significantly longer sequences compared to prior methods.

We also emphasize that training at this large input scale is inherently more challenging and unstable than training on shorter sequences. To address this, we designed a multi-resolution training strategy with combined loss functions, enabling stable training at large scales. To the best of our knowledge, this is the first feed-forward 3D reconstruction solution demonstrated at this large input scale with up to 32 images at a resolution of 960x540. Furthermore, we conducted a comprehensive study and analysis of various design choices at the same large input scale; we hope this provides helpful insights into the trade-offs and performance of different techniques for long-sequence reconstruction tasks.

We hope our work inspires the community to develop new technologies to address the challenges of the under-explored task of feed-forward wide-coverage 3D scene reconstruction.

1. Justification for using Mamba.

Ans: We do not find any claim in the GS-LRM paper about "accepting arbitrary input view numbers by downsampling images with large patch sizes to shorten the overall token length for global attention." After confirming with the GS-LRM authors, we believe they do not make this claim. Specifically, GS-LRM paper only shows results with no more than six input views. Our experiments in Tab. 2 also demonstrate that scaling GS-LRM to handle our large input scale is impractical.

Note that GS-LRM adopts a similar multi-resolution training strategy, from 256x256 low-resolution to 512x512 high-resolution images, and uses the same patch size of 8 across these resolutions. For our comparisons in Table 2, we adopted the same resolution and patch settings as GS-LRM for the two stages of training in the 2nd and 3rd blocks. We believe this ensures a fair comparison by reusing most of the hyperparameters selected by GS-LRM. With GS-LRM’s same patch size, the use of Mamba is critical for scaling up to longer input sequences.

Additionally, we found that smaller patch sizes generally yield better reconstruction quality. For example, increasing the patch size to 16 for GS-LRM with four input views (the first setting in Table 2) results in a PSNR of 20.85, lower than the original PSNR of 21.13 with a patch size of 8. We retrain using GS-LRM’s original patch size of 8, ensuring both fair comparisons and high-quality results.

2. Justification for abandoning 3D inductive biases like cost volumes.

Ans: GS-LRM has demonstrated the effectiveness of transformer-based models without relying on cost volumes or other 3D inductive biases. Building on this foundation, our work aims to scale LRM models to support long input sequences, which we believe is a promising and important direction for addressing the problem of large scene-level 3D reconstruction.

We also found that global attention methods, including both GS-LRM and Long-LRM, can significantly outperform cost volume-based methods. To illustrate this, we present a quality comparison on the RealEstate10k dataset with the standard two input views in the table below. This dataset is widely used by prior works, and the baseline results are taken directly from their respective papers. We retrained two versions of Long-LRM (w/ and w/o token merging) on the same training dataset for this comparison. The results show that global attention methods, both GS-LRM and Long-LRM, achieve significantly better performance compared to other baselines with more 3D inductive biases, including the cost volume-based MVSplat and the epipolar line-based pixelSplat.

Rendering quality on RealEstate10k with 2-view input

Additionally, we argue that cost volumes are not inherently memory-efficient for long sequences and high resolutions. Naively, one needs to construct a 3D high-resolution cost volume for every input image, requiring aggregating features from all other co-visible images, and then fuse the volumes to produce consistent Gaussians for the entire scene. This approach is computationally intensive and memory-demanding for large-scale scenes. Notably, previous feed-forward GS methods employing cost volumes (e.g., MVSplat, MVSGaussian) have only demonstrated results at 256 resolution with 1–4 input views. Successfully scaling up these models to handle long-sequence inputs remains a challenging research problem rather than a straightforward engineering task.

Overall, the task of feed-forward reconstruction for wide-coverage large scenes is still a new and underexplored area. Our work provides the first solution to this challenge, but we do not claim that global attention or Mamba-based models are the only possible solutions. It is impractical to explore every alternative in a single submission. We look forward to future explorations in this direction, including advanced cost volume-based methods and lightweight network designs, to tackle this challenge.

3. Submission to a computer vision conference vs ICLR.

Ans: ICLR has consistently served as a great venue for works at the intersection of machine learning and its applications, including computer vision. Our work falls into this category, advancing machine learning methods for 3D computer vision. Notably, several foundational papers on 3D large reconstruction models, such as LRM, Instant3D, PF-LRM, and DMV3D, have been published at ICLR, with three of them recognized as Oral or Spotlight papers (full paper titles listed below). They all address similar tasks in 3D reconstruction and generation, demonstrating the strong relevance of this area to the ICLR community. As a follow-up to the LRM series of ICLR papers, our Long-LRM tackles the new challenge of processing long-sequence inputs for wide-coverage scene reconstruction using novel machine learning techniques, which is essential for scaling feed-forward 3D reconstruction to large scenes. We believe this aligns well with the scope of ICLR and hope our work will benefit the community.

• LRM: Large Reconstruction Model for Single Image to 3D, ICLR 2023 (Oral)
• Instant3D: Fast Text-to-3D with Sparse-view Generation and Large Reconstruction Model, ICLR 2023
• PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction, ICLR 2023 (Spotlight)
• DMV3D: Denoising Multi-view Diffusion Using 3D Large Reconstruction Model, ICLR 2023 (Spotlight)

1. Technical novelty.

Ans: We emphasize that our work targets a new and under-explored task: achieving feed-forward, wide-coverage 3D scene reconstruction from a large number of input images. Prior to our work, large scene-level NeRF or 3DGS reconstructions could only be achieved by per-scene optimization methods. Recent feed-forward methods, such as GS-LRM, Gamba, and MVGamba, have made progress but focus on object-level reconstruction or sparse input views, leaving wide-coverage scene reconstruction largely unexplored. Our work is the first to demonstrate that feed-forward models can be effectively designed to handle large-scale scene reconstruction tasks, which we believe provides valuable insights to the community.

A key aspect of our contribution is addressing the challenge of long-sequence inputs with up to ~250K tokens—a scale more than ten times larger than existing transformer- or Mamba-based 3DGS models. This significantly longer sequence introduces unique challenges in maintaining high reconstruction quality while managing computational efficiency.

To tackle these challenges, we propose a novel hybrid architecture that combines transformers and Mamba2 blocks, achieving a balance between quality and efficiency. While similar hybrid architectures have been applied to other vision tasks, we are the first to adopt this hybrid approach for 3D reconstruction. In addition, we introduce token merging and Gaussian pruning techniques to further reduce computational complexity, enabling our method to handle significantly longer sequences compared to prior methods.

We also emphasize that training at this large input scale is inherently more challenging and unstable than training on shorter sequences. To address this, we designed a multi-resolution training strategy with combined loss functions, enabling stable training at large scales. To the best of our knowledge, this is the first feed-forward 3D reconstruction solution demonstrated at this large input scale with up to 32 images at a resolution of 960x540. Furthermore, we conducted a comprehensive study and analysis of various design choices at the same large input scale; we hope this provides helpful insights into the trade-offs and performance of different techniques for long-sequence reconstruction tasks.

We hope our work inspires the community to develop new technologies to address the challenges of the under-explored task of feed-forward wide-coverage 3D scene reconstruction.

2. Discussion on publicly available Mamba-based 3D reconstruction models.

Ans: We thank the reviewer for pointing out the missing citations. We will include and discuss these relevant works—Hamba, Gamba, and MVGamba—in a revision. In particular, Hamba is a Mamba-based method specifically designed for hand reconstruction, while Gamba and MVGamba utilize Mamba for feed-forward 3D GS modeling, which aligns more closely with our task. However, these methods adopt pure Mamba architectures and focus on object-level sparse-view reconstruction, typically using only 1–4 input images and processing around 10K patch tokens. In contrast, Long-LRM is designed for scene-level reconstruction from a significantly larger number of input images (up to 32) with higher resolutions, presenting key new challenges in processing long-sequence input tokens. Instead of relying on a pure Mamba architecture, Long-LRM employs a hybrid architecture that combines transformer and Mamba2 blocks, enabling efficient processing of up to 250K tokens. This scale is far beyond what previous Mamba-based methods handle. Our contributions specifically target the scalability and the unique requirements of large scene-level reconstructions with long-sequence inputs, which go beyond the scope of methods like Gamba, MVGamba, and Hamba.