MVGamba: Unify 3D Content Generation as State Space Sequence Modeling

Thank you for your insightful and positive comments! Below, we provide a point-by-point response to address your concerns. We welcome further discussion to improve the clarity and effectiveness of our work.

Q1: The performance is continuously increasing as the token length increase.

A1: The experimental results in Figure 6(b) confirm that performance improves as the token length increases. This is one of our motivation to directly process a sufficiently long token length for better performance. Consequently, we advocate for a Mamba-based architecture, which boasts linear complexity, over transformer-based architectures characterized by quadratic complexity to properly balance the performance and the computational cost, as illustrated in Table 1 in the rebuttal PDF. Furthermore, our architectural design incorporates various light-weight design elements, as described in the manuscript and detailed in Appendix C, to further reduce the computational cost and energy consumption, thereby promoting environmental sustainability in line with the growing emphasis on green AI in the Neurips society.

Q2: No failure cases are shown to demonstrate the limitation of this work.

A2:

• We have illustrated the failure cases in the Appendix D. As shown in Figure 8, MVGamba may sometimes fail if the depth of the front view is estimated incorrectly. Fortunately, this issue can be largely mitigated by manually changing the input order, as the side view typically contains sufficient depth information.

• Due to limited space in the rebuttal PDF, we couldn't provide additional failure cases discussed in the Limitations section, particularly those caused by severely flawed input views generated by the imperfect multi-view diffusion model. It's worth noting that, compared to other baselines, our results demonstrate greater robustness against inconsistent inputs due to the self-refining nature of MVGamba, which aligns with the experimental results presented in Figure 6(a) of the manuscript. We will incorporate more failure cases and their analysis in the revision as per your advice.

Q3: Fig 2(b) may be wrong. Mamba has linear complexity instead of constant complexity.

A3: We apologize for the confusion. We confirm that Figure 2(b) is correct; it indeed illustrates linear complexity. Due to the quadratic complexity of the Transformer, we had to scale the y-axis to prevent the Transformer's curve from extending outside the figure. We will revise this figure to avoid confusion in revision.

Q4: The video results in the supplementary materials are only marginally better than LGM's results.

A4: We believe that our proposed MVGamba could consistently outperform LGM in most cases. To address your concerns, we have provided more qualitative results compared to LGM in Figure 3 in the rebuttal PDF, where MVGamba could generate more accurate geometry and fine-grained textures, while LGM exhibits blurred texture and severe multi-view inconsistency, eg., multiple eyes on the dragon's face.

Dear Reviewer 5iTN,

Thank you so much for your response and support. Regarding Figure 2, we observe that as the token length increases from 1024 to 16384, the theoretical FLOPs for transformer self-attention rise dramatically from 2.15 to 292.06, while Mamba's SSM demonstrates a linear increase in FLOPs, scaling from 0.07 to 1.07 as shown in Table 1 of the rebuttal PDF. Plotting these together necessitates scaling the y-axis, which could potentially cause confusion by making Mamba's complexity appear constant, which indeed is linear. Per your suggestion, we will clearly indicate the y-axis scale and specific FLOPs values to prevent misinterpretation in the revision.

If you have any further concerns or questions, welcome to discuss with us. We are more than happy to discuss with you further and provide additional materials.

Once again, thank you for your valuable time and sound advice!

Best regards,

Authors of Submission 2178

Thank you for your positive feedback and insightful comments! Below, we provide a response to address your concern. We welcome further discussion to enhance the clarity and effectiveness of our work.

Q1: How does a non-pixel-aligned transformer model perform?

A1:

• Non-pixel-align is not the primary cause. Per your suggestion, we tested a non-pixel-aligned transformer model for 3DGS prediction. As discussed in our overall response, those methods typically process a compressed sequence due to the computational budget. We followed OpenLRM[1] to model 4096 tokens, and adopt a de-convolution layer to up-sample from 4096 to 16384. However, as shown in the rebuttal PDF Figure 2, it yielded inferior performance, with broken geometry and blurred textures. We attribute this to the inherent non-convex optimization challenge in the inverse graphics scenario, as mentioned in pixelSplat[2], which may be further amplified by the de-conv upsampler. This preliminary result may also explain why pixel-aligned Gaussian are widely adopted by recent transformer-based approaches with upsamplers.

• The crux is to directly model a sufficiently long sequence of 3DGS in a causal manner. As mentioned in the manuscript and our overall response, the direct sequence modeling paradigm allows for more fine-grained detail modeling with cross-view self-refinement . Moreover, our direct sequence modeling paradigm can also ease optimization, which provides a new feasible way for training Gaussian LRMs.

[1] He, Z. and Wang, T. OpenLRM: Open-Source Large Reconstruction Models.

[2] Charatan, D., et al. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. CVPR2024.

Q2: Model size comparison.

A2: Thank you for pointing out! The following table discusses the model size that supports our claim. Our statement of 0.1× of the model size refers to a comparison with open-source SOTA models TripoSR and LGM. This table will be included in the revision.

Q3: Reconstruction results comparison.

A3: To address your concerns, we provide an additional qualitative comparison using samples derived from the original GRM paper. As shown in the rebuttal PDF Figure 3, MVGamba accurately captures better geometric structures and comparable appearance details compared to GRM. We could not directly compare with GS-LRM as it only presents a single-view results in Figure 7 of their paper. Note that while GRM and GS-LRM bring impressive generation results, they were both pre-printed on arXiv close to the NeurIPS deadline without open-source code, and even some of the multi-view diffusion models they leveraged is not available to public yet, eg., Instant3D. Additionally, both GS-LRM, with 300M parameters, and GRM, whose parameters are not disclosed, appear significantly larger than MVGamba with 49M parameters.

Q4: What does 'post hoc' operation mean?

A4: 'Post hoc' refers to the merge operation discussed in lines 39-40. For instance, GS-LRM and LGM increase the number of Gaussians by simply merging from different views. In contrast, our MVGamba can directly generate a sufficiently long Gaussian sequence for all multi-views simultaneously without such operation. Furthermore, as discussed in our overall response, recent transformer-based methods such as GRM and GS-LRM involve an upsampler to increase the number of Gaussians after modeling the patch tokens. This type of post sequence modeling operation is also not required for MVGamba.

Q5: What is the number of output 3D Gaussians? How to increase it?

A5:

• As shown in Figure 1(b) in the rebuttal PDF, the number of predicted Gaussians in MVGamba equals the number of output tokens since MVGamba directly decodes Gaussians from the generated long Gaussian tokens.

• There are several feasible ways to increase number of Gaussians. (1) Different scan orders: Due to the causal nature of MVGamba, we can apply various scan orders to expand patches, each increasing the number of tokens as well as the number of Gaussians by N. (2) Pixel-aligned Gaussian representation. Following the concurrent Gaussian LRMs, we could leverage upsampler to unpatchify the output tokens into per-pixel Gaussians. We believe this method might be more effective with higher image resolution and more accurate and consistent multi-view inputs.

Q6: Larger number of input views.

A6: We have conducted experiments with dense view settings in the sparse view task. The Table below indicates that MVGamba effectively manages 6 input views due to its computational efficiency, achieving a notable improvement in reconstruction performance in the GSO test set. Further studies with denser input views (10 views or more) will be included in the revision.

Q7: Scene-level reconstruction is an interesting future work.

A7: Due to time limitations, we were unable to demonstrate MVGamba's capacity for scene-level reconstruction during the rebuttal period. However, we note that MVGamba's computational efficiency and cross-view self-refinement gives it potentially significant advantages in scene reconstruction, which require more Gaussians and higher image resolution. Theoretically, with the introduced pixel-aligned Gaussian representation, MVGamba can operate at a resolution of 2560 $\times$ 1600, compared to 512 $\times$ 904 for GS-LRM, within a similar overall computational budget. We plan to further explore the Mamba-based reconstructor for scene reconstruction as future work.

Dear Reviewer eshR,

We wish to convey our sincere appreciation for your insightful and invaluable feedback, which has been of great help to us. As the discussion deadline approaches, we are keenly anticipating any additional comments or suggestions you may have. Ensuring that the rebuttal aligns with your suggestions is of utmost importance. We are deeply grateful for your commitment to the review process and your generous support throughout.

Best regards,

Authors of Submission 2178

Thank you for the positive and insightful comments! In the following, we provide our point-by-point response and look forward to the subsequent discussion.

Q1-1: Input tokens are short.

A1-1: 3DGS reconstruction is a long-sequence task. We analyze the token length of two paradigms in rebuttal PDF Figure 1:

• Direct long-sequence modeling: Requires directly processing more than 16k tokens.
• Compressed sequence modeling: For a standard $384 \times 384$ resolution input with a large hidden dimension $D=1024$ in GS-LRM, [1] proposed a ratio-based metric, where we derive $r_L \approx 1.5 > 1$, confirming it as a long-sequence task even with compressed sequences. Note that $r_L \approx 0.04$ in traditional image classification tasks.

Moreover, we agree that Mamba-based reconstructor may potentially offer more advantages as views and sequence length increase, which we consider an interesting direction for future work.

[1] Yu, W., & Wang, X. MambaOut: Do We Really Need Mamba for Vision?. arXiv:2405.07992.

Q1-2: Inference speed in Table 1.

A1-2:

• In Table 1, MVGamba's inference time includes both multi-view image generation and multi-view reconstruction (0.03s), as detailed in Appendix A. When considering the reconstruction module alone, MVGamba is at least $3.6\times$ faster than GRM (0.11 sec), TGS (0.2 sec), and GS-LRM (0.23 sec).

• Importantly, MVGamba's direct sequence modeling achieves significant inference speed improvement even when modeling sequences that are $4\times$ longer.

Q1-3: Necessity of Mamba.

A1-3: Mamba-based reconstructor is necessary. Given the above responses in A1-1 and A1-2, we believe: 1) Sparse-view 3DGS reconstruction is a long-sequence task; 2) MVGamba achieves at least $3.6\times$ faster inference speed while directly modeling $4\times$ longer sequences. To our knowledge, Mamba-based reconstructor is the only experimentally validated direct sequence modeling approach. As detailed in overall response, MVGamba efficiently balances computational cost with long-sequence modeling capacity and offers additional valuable advantages, making it crucial for direct sequence modeling.

Q2-1: Quantitative comparison with GS-LRM.

A2-1:

• Currently, conducting a direct and fair quantitative comparison between MVGamba and GS-LRM meets significant challenges. We believe the performance gap is largely due to differences in experimental settings and evaluation protocols, such as differences in test samples, rendered views, and camera trajectories. Notably, even the number of input views for text/image-to-3D tasks differs, as GS-LRM utilizes 6 generated images in their multi-view reconstructor. While GS-LRM is promising, it was pre-printed on arXiv near the NeurIPS deadline without released code, leaving us insufficient time for thorough analysis or reproduction. We are eager to conduct a comprehensive comparison once GS-LRM is open-sourced or re-implemented. MVGamba will also be fully open-sourced to facilitate the 3D community.

• MVGamba's reconstruction quality can be further improved with larger model size. Our newly trained MVGamba-B (110M) surpasses the original MVGamba-S (49M) in the manuscript by 1.9 dB in the GSO dataset for sparse-view reconstruction. We plan to further explore the potential of MVGamba with larger model size, advanced 3D representations and stronger Mamba-based architectures such as Mamba-2[2] and Samba[3].

[2] Dao, T., & Gu, A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. ICML2024.

[3] Ren, L., et. al. Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling. arXiv:2406.07522.

Q2-2: Is Mamba important for quality?

A2-2: Mamba is crucial for performance improvement. As detailed in A1-3, the Mamba-based reconstructor offers several unique advantages that enhances both quality and efficiency.

To further address your concerns, Figure 3 of the rebuttal PDF shows a qualitative comparison using samples from the GRM paper. It indicates that MVGamba captures better geometries and comparable details with significantly smaller model size. Note that we could not directly compare with GS-LRM as it only presents a single-view results in Figure 7 of their paper.

Q3: Why might GS-LRM suffer multi-view inconsistency?

A3: Potential multi-view inconsistency in GS-LRM may stem from:

• View-separated up-sampling: GS-LRM upsamples tokens in each view separately using local deconvolution, and a large amount of Gaussians are predicted in each isolated view without 'attention' to mutual information from other views.
• Merge operation: GS-LRM merges these separately upsampled Gaussians without multi-view context modeling.

Note that GS-LRM was pre-printed near the NeurIPS deadline without released code. All the analysis above is based on carefully reading their paper and our empirical knowledge of this field, acknowledging potential understanding limitations without access to GS-LRM's implementation details.

Q4: Why choose patch size 14?

A4: We chose a patch size of 14 based on theoretical and empirical considerations.

• Theoretical Insight: MVGamba is a non-pixel-aligned model that processes causal context, requiring each patch to contain sufficient information, similar to traditional vision transformers like ViT-H-14. In contrast, GS-LRM is a pixel-aligned model and performs per-pixel Gaussian generation may require smaller patch.

• Empirical Evidence: Ablation studies with a fixed token length of 16k on the Human-Shape subset showed patch size 14 outperformed patch size 8 by 0.71 dB PSNR. The average loss was 0.03243 for patch size 14, compared to 0.03918 for patch size 8, supporting that MVGamba benefits by patches with sufficient information.

Besides, MVGamba offers additional ways to increase sequence length. Please kindly refer to the response to eshR A5.

Dear Reviewer XU7f,

We would like to express our heartfelt gratitude for your insightful and invaluable comments, which have been of great help to us. As the discussion deadline approaches, we are eagerly looking forward to receiving your valuable feedback and comments. Ensuring that the rebuttal aligns with your suggestions is of utmost importance. Thank you again for your dedication to the review process and your generous support.

Best regards,

Authors of Submission 2178

Dear Reviewer XU7f,

Thank you very much for your comments and support! It is highly encouraging to learn that our rebuttal has successfully addressed the majority of your concerns. We would like to take this opportunity to discuss two additional points with you:

Sequence Length and computational cost. We emphasize that the Mamba-based reconstructor not only offers a speed advantage over GS-based methods but also, to our knowledge, currently represents the only experimentally validated direct 3DGS sequence modeling approach capable of processing over 16K tokens, as discussed in our overall response. We also observe that with higher resolution inputs, there shows a substantial computational cost (heavy GPU memory) even for compressed sequence modeling, evidenced by the fact that with 512 resolution inputs, the batch size of GS LRM is 2, which is much smaller compared to 8 or 16 in regular LRM experimental settings, even with several efficient training strategy to save GPU memory, eg., gradient checkpointing, deferred backpropagation and mixed-precision training with BF16.

Visual Quality. We believe that MVGamba could achieve comparable visual quality compared to concurrent GRM and GS-LRM with extremely smaller model size. In our future work, we intend to further enhance the reconstruction quality and explore the full potential of MVGamba through several avenues: increasing the model size, integrating more advanced multi-view diffusion models, and incorporating stronger Mamba-based architectures.

We welcome any further concerns or questions you may have and are eager to engage in additional discussions or provide supplementary materials.

Once again, we express our sincere gratitude for your recognition of our work and your invaluable review feedback.

Best regards,

Authors of Submission 2178