LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias

Thank you for your insightful comments and valuable suggestions. We have revised our paper based on your feedback. Here are our responses to your comments:

1. Questions related to view consistency

   As demonstrated in the video results on our project webpage, our generalizable deterministic models, trained on large-scale multi-view data, are capable of generating high-quality novel view synthesis results while maintaining strong view consistency. This demonstrates that 3D reconstruction or explicit 3D representations are not strictly necessary for achieving view consistency. View consistency can be learned by transformer architecture. The advancements of recent video diffusion can also support this observation.

   For the evaluation of novel view synthesis, we have shown PSNR metrics for the reprojected novel view images for both object and scene, as shown in Table 1. PSNR measures the pixel-level error between our prediction and the ground-truth images, which can work as a good evaluation for the 3D consistency, as done in prior NeRF/3DGS methods.

2. Detailed Diagram

   We sincerely thank you for your suggestion. We have revised our paper and included a detailed model architecture diagram in Figure 8 of the Appendix. Hope this can help readers better understand our models. Please let us know if something is still unclear.

3. Questions regarding the qualitative results are not consistent with the quantitative results.

   Although our encoder-decoder model and GS-LRM achieve comparable PSNR results, our encoder-decoder model clearly has better perception-based LPIPS metrics performance, which aligns better with visual quality than PSNR. If the reviewer remains concerned about this issue, we are happy to include additional qualitative results in a future revision of the paper.

4. Adding failure cases

   We appreciate the reviewer's suggestion regarding failure cases. In response, we have added a dedicated limitations section in Appendix A7 and included related failure cases on our anonymous project webpage.

5. Efficiency comparison

   We do not claim that our methods are more efficient than previous approaches. We sincerely apologize for any misunderstanding this may have caused. If the reviewer can point out the specific sections in question, we would be happy to revise the corresponding descriptions.

6. Question related to code release

   We will release our code in the near future.

Thanks for the author(s)' response. Here are some further comments:

1. The concern has been well addressed. I believe that the view consistency is properly learned in the transformer after seeing the examples with unseen regions, where the proposed LVSM is consistent at seen regions but suffers from minor jittering at the unseen parts.
2. The diagram is clear and easy-to-understand now.
3. The claim is reasonable and accepted, while it's better to point out what kind of examples results in better LPIPS but worse PSNR.
4. The concern has been well addressed.
5. Generalizable models are proposed to increase the efficiency of novel view synthesis and 3D reconstruction. In the 075 line, the author(s) also mentions how they contribute to a scalable and efficient novel view synthesis model. However, I cannot see an obvious motivation about removing 3D inductive bias, if the model increases much computational complexity with comparison to other reconstruction-based large models, e.g. LRM, GS-LRM. Specifically, I'm asking what the largest advantage of removing 3D inductive bias is. This includes but is not limited to:
   • Is LVSM easier to be trained than GS-LRM? (more stable or faster)
   • Does LVSM supports faster rendering than other LRMs? If the efficiency is not you major claim, it's better to fix the over-claim in 075 line. Also, how much speed sacrifice is needed, compared to reconstruction-based generalizable models? Does LVSM still support real-time rendering?
6. Thanks.

I may temporarily lower my score before the (5.) comment is properly addressed.

Thank you for your additional comments.

We are genuinely pleased to see that our responses have addressed most of your concerns, except for point 5. Here, we provide further clarification:

• Training Stability and Efficiency: We found that LVSM is easier to train compared to GS-LRM, with training being more stable and faster. Specifically, during training, the final LVSM with QK-norm generally exhibited a lower gradient norm than GS-LRM with QK-norm, making it much less likely to crash. As said in GS-LRM's paper on page 21, it needs to design a GS initialization carefully with some magic number to help training stability, while LVSM doesn't need to. Additionally, as noted in our Lines 370-410, LVSM trained on 2 GPUs outperformed GS-LRM trained on 64 GPUs, further demonstrating its training efficiency.

• Rendering Efficiency: We want to emphasize that we do not intend to claim our model has better rendering efficiency compared to GS-LRM. As shown in Figure 6, we reported our rendering FPS, and we acknowledge that GS-LRM benefits from the explicit GS representation, resulting in better rendering efficiency.

To clarify this, we have revised the term "efficient" in Line 75 to "training-efficient.", and we will further clarify this in the final revised paper.

We sincerely apologize for any misunderstanding this may have caused. As the discussion period deadline approaches, please don't hesitate to share any further concerns. We hope that our responses resolve all your concerns, and we kindly hope you can restore the rating.

Thanks for the response. The concern in comment (5.) has been resolved.

Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:

1. Clarification of the novel elements of this architecture compared to prior work

   Our task is to synthesize novel views from sparse views. Previous SRT shares a similar task as ours, and it also doesn’t have handcrafted 3D representations. However, our two proposed models, Encoder-Decoder LVSM and Decoder-Only LVSM are highly different from SRT. The details are as follows:

   (1). Our encoder-decoder LVSM is similar to SRT (Sajjadi et al., 2022) at a high level – Both SRT and our encoder decoder leverage an encoder to transform input images into a set of 1D tokens that serve as an intermediate latent representation of the 3D scene. A decoder can then render novel view images from this latent representation. However, encoder-decoder LVSM introduces a highly different architecture that significantly improves performance. We have provided related ablation study experiments and a more detailed analysis in our revised paper. (L428-L470, and Table 3). We summarized the key differences as follows:

   • (a). We used a simpler and more effective patch-based input image tokenizer, which improves the performance, compared with SRT’s CNN-based tokenizer.
   • (b). Our encoder progressively compresses the information from posed input images into a fixed-length set of 1D latent tokens. This design ensures a consistent rendering speed, regardless of the number of input images, as shown in Fig. 6. In contrast, SRT’s latent token size grows linearly with the number of input views, resulting in decreased rendering efficiency.
   • (c). The decoder of our encoder-decoder model utilizes pure (bidirectional) self-attention, which enables i) latent tokens to be updated across different transformer layers, which also means the parameters of the decoder are a part of the scene representation; ii) output patch pixels can also attend to other patches for joint updates, ensuring the global awareness of the rendered target image. Prior work SRT (Sajjadi et al., 2022) has a cross-attention-based design for its decoder, which doesn’t support these functions. We experiment with an LVSM variant by adopting SRT’s decoder designs, which leads to significant performance degradation, as shown in Table 3 of the revised paper.

   (2). Our decoder-only LVSM further pushes the boundaries of eliminating the inductive bias and bypasses any intermediate representations. It adopts a single-stream transformer to directly convent the input multi-view tokens into target view tokens, treating the view synthesis like a sequence-to-sequence translation task, which is fundamentally different from previous work, including SRT.

   (3). In addition, we have implemented several design choices to make our scalable and effective training happen, including predicting patch pixels instead of ray pixels for more efficient rendering, using better camera pose information imbedding, integrating Flash-Attention, Gradient Checkpointing, and mixed precision training for more efficient training, and using QK-norm for more stable training. The technical designs such as Flash-Attention and Gradient Checkpointing are important for final high-quality results but are often overlooked in the 3D community.

2. The presentation is messy

   We sincerely appreciate your suggestions for clarifying our presentation. We have rewritten Sec.3.2 to make it more clear and concise. We have clarified the model size information by providing the model parameter counts in Table 2. We also make the computing information more clear.

3. Have Encoder-Decoder VS Decoder-Only on both Object- and Scene-Level

   We have updated our main paper and provided both the scene- and object-level results in Table 2. The object and scene have observed a similar scaling tendency.

Thank you for your insightful comments and valuable suggestions. We have revised our paper based on your feedback. Here are our responses to your comments:

1. More illustrations for the architecture

   Thank you for your suggestion. We have now included a detailed model architecture diagram in Figure 8 of the Appendix.

2. Clarification about the decoder architecture

   (Note: Due to paper edits during rebuttal, the original “L303–308” is now “L262–269”) (a). Yes, we use the same self-attention block for all encoder-decoder and decoder-only layers. (b). As said in L889 and L893, for the scene-level experiment, we use 2 input views and 6 target views for each training scene. For the object-level experiment, we use 4 input views and 8 target views for each training object. During both training and inference, our model predicts each target image one at a time. We also experimented with predicting multiple target views simultaneously in a single feedforward pass with our pre-trained model. However, this led to degraded performance. For instance, on the GSO (res 256) dataset, the decoder-only model’s performance dropped as follows:

   • PSNR: From 31.71 to 29.61
   • LPIPS: From 0.027 to 0.039
   • SSIM: From 0.957 to 0.943

   This degradation is understandable, as the model is trained to predict only one target image per feed-forward pass, and deviating from this setup during inference affects its performance.

Thank you for your insightful comments and valuable suggestions. We will revise our paper based on your feedback. Here are our responses to your comments:

1. Difference between SRT and LVSM

   We respectfully disagree with the assertion that “our proposed idea is a very specific, minor change to SRT”. Prior SRT shares a similar task as ours, and it also doesn’t have handcrafted 3D representations. However, our two proposed models, Encoder-Decoder LVSM and Decoder-Only LVSM are highly different from SRT. Importantly, we never claimed that the removal of CNNs is the key factor driving the superior performance of our models. In fact, our models have numerous design differences from SRT, with the removal of CNNs being just one minor component. We apologize for any confusion and will clarify this further as follows:

   (1) Our Encoder-Decoder LVSM is similar to SRT (Sajjadi et al., 2022) at a high level (both use an encoder to transform input images into a set of 1D tokens serving as a latent representation of the 3D scene, which is then decoded to render novel views). However, Encoder-Decoder LVSM introduces a highly different architecture that significantly improves performance. We have provided related ablation study experiments and a more detailed analysis in our revised paper. (L442-L470, and Table 3). We also summarized the key differences as follows:

   • (a). Simpler and more effective tokenizer: We used a simpler and more effective patch-based input image tokenizer, which improves the performance, compared with SRT’s CNN-based tokenizer. (Notably, although removing CNN here brings performance improvement, this change brings less obvious and less significant performance differences ompared with our other design differences, as shown in Table 3 of the revised paper. We will discuss other design differences later. Therefore, we respectively disagree that our contribution is just "replace the original CNN". In addition, we would like to point out that SRT employs CNNs only as a input tokenizer and is otherwise pure transformer-based. )
   • (b). Progressive compression for fixed-length latent tokens: Our encoder progressively compresses the information from posed input images into a fixed-length set of 1D latent tokens. This design ensures a consistent rendering speed, regardless of the number of input images, as shown in Fig. 6. In contrast, SRT’s latent token size grows linearly with the number of input views, resulting in decreased rendering efficiency.
   • (c). Joint updating of the latent and target patch tokens: The decoder of our encoder-decoder model utilizes pure (bidirectional) self-attention, which enables i) latent tokens to be updated across different transformer layers, which also means the parameters of the decoder are part of the scene representation; ii) output patch pixels can also attend to other patches for joint updates, ensuring the global awareness of the rendered target image. Prior work SRT (Sajjadi et al., 2022) has a cross-attention-based design for its decoder, which doesn’t support these functions. We experiment with an LVSM variant by adopting SRT’s decoder designs, which leads to significant performance degradation, as shown in Table 3 of the revised paper (this single change will make PSNR decrease ~3.5dB ). We have also ablated each of the above two functions mentioned individually, showing their effectiveness, and discussed this in detail from L458-469.

• Our Decoder-Only LVSM further pushes the boundaries of eliminating the inductive bias and bypasses any intermediate representations. It adopts a single-stream transformer to directly convent the input multi-view tokens into target view tokens, treating the view synthesis like a sequence-to-sequence translation task, which is fundamentally different from previous work, including SRT.

• In addition, we have implemented many other design choices to make our scalable and effective training happen, including using plucker rays to better represent the camera information, predicting patch pixels instead of ray pixels for more efficient rendering, integrating Flash-Attention, Gradient Checkpointing, and mixed precision training, for more efficient training, using QK-norm for more stable training, etc. These technical designs are important for final high-quality results.

Unfinished. Please keep reading the comments.

7. Extrapolation results

   The anonymous project webpage has already included pose extrapolation video results on object-level data, in which our methods have a good NVS performance.

   We have also updated our anonymous project webpage and provided some extrapolation results for scene data. Given two input images with poses 1 and 2, we generate a novel view rendering video that transitions through poses 0 → 1* → 2* → 3. This means the beginning and end of each video involve pose extrapolation, while the middle segments interpolate between poses 1 and 2. Please visit the anonymous project webpage for visual results.

   As discussed in the limitations section of the revised paper, our models are deterministic. Therefore, similar to all prior deterministic approaches (e.g., MVSNeRF, IBRNet, SRT, PixelSplat, LRMs, GS-LRM), our models, struggle to produce high-quality results in unseen regions. When the camera moves into unseen areas, the rendering quality degrades, which is often noticeable at the start of the following videos. However, when extrapolation occurs within regions covered by the input views, the rendering quality remains good, as often observed near the end of the following videos. This behavior is also consistent with the observations from single-view synthesis results. We will illustrate this more clearly in the revised paper.

   That said, we found our methods still have better quantitative extrapolation results compared with our baselines. Recent work latentSplat[1] (ECCV 2024) created a testing split of Realestate10k designed for extrapolation testing. We test our extrapolation results on its testing dataset

   Method	PSNR↑	SSIM↑	LPIPS↓
   pixelNeRF [2]	20.05	0.575	0.567
   Du et al. [3]	21.83	0.790	0.242
   pixelSplat [4]	21.84	0.777	0.216
   latenSplat[1]	22.62	0.777	0.196
   Ours Encoder Decoder	24.86	0.827	0.164
   Ours Decoder Only	26.18	0.857	0.140

   As shown above, our methods perform much better than the baseline in the extrapolation setting. The baseline results here are copied from latentSplat[1] (ECCV 2024)’s Table 2. (We didn’t test MVSplat and GS-LRM here because we have received a lot of reviews and we only have limited time. But we are happy to incorporate them or other related papers suggested by the reviewer in our final revision if the reviewer thinks this is important.)

   [1] latentSplat: Autoencoding Variational Gaussians for Fast Generalizable 3D Reconstruction (ECCV 2024)

   [2] pixelNeRF: Neural Radiance Fields from One or Few Images (CVPR 2021)

   [3] Learning to render novel views from wide-baseline stereo pairs (CVPR 2023)

   [4] pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (CVPR 2024)

2. Comparison with SRT

   SRT’s current implementation does not support efficient large-scale training, making it difficult to train SRT on our dataset in a computationally efficient and stable manner. Additionally, as we mentioned in our previous response, the architectural differences between our encoder-decoder model and SRT are not small but significant. They are similar at a high level, but many detailed designs are different, including different input tokenizers, different transformer encoder designs, different latent representations, different transformer decoder attention designs, different camera pose representations, different pixel rendering designs (single ray-based Versus multi-patch-based), etc. Therefore, rather than modifying SRT’s code to work with our dataset, we believe a more meaningful comparison is achieved through ablation studies where we replace our encoder-decoder LVSM model components with SRT’s design. This allows us to isolate the other factors, and better demonstrate and understand the effectiveness of our architectural design choices.

   As shown in the ablation studies that are discussed earlier, our design choices for our encoder-decoder LVSM led to clear performance differences, which we believe provides strong evidence to prove our better performance is not just because of the data we used. We sincerely appreciate your detailed feedback, and we have revised the main paper to include a more detailed discussion and a stronger claim regarding these points in response to your feedback(see L428–L481).

3. Difference compared with previous Diffusion based methods

   Our generalizable deterministic methods are fundamentally different from generative models (including diffusion-based models). We have provided a detailed discussion about the difference between our models and the generative models in Appendix A.6 of our revised paper, which has included all the papers you mentioned. If you have further questions or would like additional clarification, we would be happy to provide more detailed responses.

4. More scene-level visual results

   We appreciate your suggestion of evaluating our model on more complex datasets. In response, we have updated our anonymous project page to include our model's results on the other cross-domain view synthesis datasets, such as MipNeRF360 and LLFF. Please refer to the webpage section titled “Results on Cross-Domain Datasets” for more details. Our model achieves reasonably good results on these datasets.

   However, we would like to highlight that it is uncommon to use those datasets you mentioned to evaluate sparse-view synthesis models trained on RealEstate10k. For instance, our scene-level baselines—including PixelSplat (CVPR 2024, Oral, Best Paper Runner-Up), MVSplat (ECCV 2024, Oral), and GS-LRM (ECCV 2024)—are all only validated on RealEstate10k or ACID (which is simpler than RealEstate10k).

   More complex datasets like MipNeRF360 differ significantly from RealEstate10k in terms of data distribution, making them less suitable for evaluating our model. Key differences include:

   • Camera baseline distances: These model complex datasets involve larger baseline distances between input views.
   • Camera trajectory distribution: RealEstate10k predominantly features forward or backward camera motion, while those more complex datasets include more different and diverse camera motion trajectories. For example, MipNeRF360 mainly includes 360-degree motion tracks.
   • Camera intrinsic parameters: Differences include variations in field of view (FOV), aspect ratio, and other intrinsic properties.

   These datasets are more suited for validating view synthesis models trained with denser input views (such as recent LongLRM) and trained on datasets with similar distributions, such as DLV3D.

5. Failure cases

   We sincerely appreciate your suggestion, and we have now provided a limitation section in the Appendix. A7. And we have updated our anonymous project page to show failure cases.

6. Question-related to 3D consistency

   As demonstrated in the video results on our project webpage, our generalizable deterministic model, trained on large-scale multi-view data, generates high-quality novel view synthesis results with strong view consistency. As we answered previously, unlike probabilistic generative models (e.g., diffusion models), our deterministic approach doesn’t have the inherent view inconsistency issues that are often associated with probabilistic generative frameworks.

   In our revised paper, we have included a detailed discussion on the differences between our model and generative models ,and the view consistency problem. Please refer to Appendix A.6 for more details. And we are happy to discuss if you have further questions.

Unfinished. Please keep reading the comments.

Thanks for your further feedback. We explain you left questions as follows:

1. More Explanation of the 3D Consistency

   Our methods are not only visually consistent but also demonstrate strong 3D consistency:

   • We have reported PSNR metrics for both object-level and scene-level novel view synthesis results, as shown in Table 1. PSNR measures the pixel-level error between our predictions and the ground-truth images, serving as a reliable evaluation metric for 3D consistency, as established in prior NeRF/3DGS methods.
   • Our results are also consistent enough to reconstruct 3D mesh from it. For example, in the “Results on Cross-Domain Datasets” section of the updated anonymous webpage, we demonstrate our method’s ability to render consistent view synthesis results using four sparse input views from cross-domain object-level datasets (e.g., the hotdog example from the NeRF synthetic dataset). Using these synthesized views, we successfully reconstruct a 3D mesh with good quality using common 3D reconstruction methods, such as the recent 2DGS. You can view the reconstructed mesh for the hotdog example here: https://3dviewer.net/#model=https://lvsm-web.github.io/mesh/hotdog.ply . If needed, we are happy to include more results to further support our claims.

2. Cross-Domain Generalization Evaluation

   • As explained in our previous reply (Point 4), we have updated our anonymous project page and included our model's results on other cross-domain view synthesis datasets, such as NeRF-Synthetic, MipNeRF360, and LLFF, to show our methods' generalization. Please refer to the webpage section titled “Results on Cross-Domain Datasets” for more visual results.

   • For the quantitative evaluation, to start with, we would like to clarify that previous work, including pixelSplat and MVSplat, did not train their models on the RealEstate10k dataset and then evaluate them on ACID. Instead, these methods were trained directly on ACID and evaluated on ACID. (In their official repositories, they have released separate checkpoints for Realestate10k and ACID.)

     Our models can outperform prior methods on ACID, even when solely being trained on Realestate10k. The results are as follows: (We test GS-LRM and our methods by ourselves, and report results for other methods based on their paper.)

     Method		ACID
     	PSNR ↑	SSIM ↑	LPIPS ↓
     pixelNeRF *	20.97	0.547	0.533
     GPNR*	25.28	0.764	0.332
     Du et al. *	26.88	0.799	0.218
     pixelSplat*	28.27	0.843	0.146
     MVSplat*	28.25	0.843	0.144
     GS-LRM	28.84	0.849	0.146
     Ours Encoder-Decoder	29.05	0.846	0.159
     Ours Decoder-Only	30.44	0.869	0.126

     (Note, ”*” here denotes methods trained with ACID dataset)

     As shown in the table above, our methods—trained only on RealEstate10k—achieve better or comparable overall performance on ACID dataset compared to all prior methods, including those explicitly trained on the ACID dataset (e.g., pixelSplat and MVSplat).

     This demonstrates that our methods exhibit superior performance and generalization capabilities.

We hope the explanation here addresses your concerns. We sincerely appreciate your valuable efforts and feedback.

Thank you for your insightful comments and valuable suggestions. We have revised our paper based on your feedback. Here are our responses to your comments:

1. Adding discussions on how the methods differ from generative mult-view methods

   We sincerely appreciate your suggestion of adding some proper discussions of how our methods differ from the generative methods, including Free3D and EscherNet. We have added a detailed discussion of it in Appendix A6 of our revised paper, and we now have also cited all those papers you mentioned in our “Related Works”.

2. Why do we choose to use latent tokens with fixed length

   Using uncompressed reference tokens does indeed result in linear complexity. However, since we employ bidirectional self-attention in all transformer layers, a linear increase in the transformer's input token size leads to a quadratic growth in computational cost. Consequently, this significantly reduces rendering speed (FPS).

3. If the attention is fully bi-directional and if introducing asymmetric masking strategies can help improve performance

   • Yes, our current models only use fully bidirectional self-attention blocks for all transformer layers. In L 458-469, we have shown some relevant ablation studies for the masking strategies. The results show that fully bi-directional self-attention achieves the best results across different attention mask designs.
   • We only predict one target image for each single transformer pass. The target images are attained from the initial target pose tokens, and the loss is only computed on the target image. We have provided a detailed diagram in our revised papar (Fig.8). Please let us know if you have additional questions.

4. Unified Model Training

   Unified Model Training is an interesting point and we appreciate you mentioning this.

   During the rebuttal period, we tried unified training by mixing the object data and scene data together in the same dataloader. The data loader randomly fetches scene data or object data during training. We also increased the input view number for the scene from the original number 2 to 4, to align with the object data (object data has 4 input images), so that mixed object and scene data can be fed into the the same batch during training. Due to the time limit, we only our decoder-only model under 256 resolution, using the same training configuration (including iteration num, batch size, etc ) as the original scene-level setup. The final model performance is:

   	PSNR	SSIM	LPIPS
   Realestate10k	28.81	0.894	0.107
   GSO	30.43	0.947	0.044
   ABO	31.12	0.937	0.056

   Based on the above results, we can see that the model can be jointly trained on both object and scene data, enabling it to handle these two data types within a single framework. This demonstrates its potential as a more general-purpose view synthesis framework capable of managing both object and scene data simultaneously.

   While the jointly trained model exhibits a 0.8–1.3 PSNR performance drop compared to models trained exclusively on either objects or scenes, this is understandable for two key reasons:

   • Handling both object and scene data in a unified model is inherently more challenging, likely requiring additional model capacity and computational resources.
   • Due to time constraints, we have not yet explored the optimal training configurations, leaving room for further optimization.

5. Single-view synthesis comparison

   We want to emphasize that single-view synthesis is not the primary focus of this project. We include some single-view synthesis results solely to demonstrate that our model has a certain level of 3D understanding, such as depth understanding, rather than focusing on pixel-level view interpolation. We will clarify this point more explicitly in the final revised version of the paper.

   To ensure a fair comparison with methods specifically designed for single-image scenarios, we would need to retrain our model using a redesigned training setup to handle single-image inputs. However, this falls outside the scope of our project's focus and cannot be achieved within the limited time frame of the rebuttal period.

   Furthermore, we believe that single-view synthesis inherently involves posing scale ambiguity, making fair quantitative comparisons particularly challenging. Addressing this would require additional time to design a fair evaluation setup, which is also beyond the scope of this project.

Unfinished. Please keep reading the comments.

Thanks for your further feedback.

For point 2, we understand the naming of the decoder can be a little confusing for some readers. In our original paper, we used a separate paragraph in Sec. 3.2 to explain this. During the rebuttal period, we received some feedback from another reviewer for this issue, and we revised our paper based on the reviewer's suggestion and now mention the decoder naming in L239 and Appendix A1. We sincerely appreciate your suggestion and will further clarify this point in the final paper based on your suggestion.

For point 9, the input views are sampled in an object-centric manner on a sphere with a fixed view radius. But the rendered novel view synthesis video results are sampled with varying radius (from 1.8 to 3.2), which makes us think the object results are extrapolation.

Thank you for your insightful comments and valuable suggestions. We have revised our paper based on your feedback. Here are our responses to your comments:

1. LVSM is not a ‘reimplementation’ of SRT

   We respectfully disagree that “LVSM seems to be a ‘reimplementation’ of SRT with more recent modules”. It’s true that both SRT and LVSM share the same motivation – removing 3D inductive bias, which was first explored in Light field networks[1](NeurIPS 2021), but our models (Encoder-Decoder LVSM and Decoder-Only LVSM) have highly different architecture designs from SRT. To illustrate, Encoder-Decoder LVSM and SRT both have encoder-decoder designs, but this only makes them similar at a high level and LVSM has a lot of different architecture designs, which significantly improve our performance. (More details will be discussed in point 2 of this rebuttal reply). In addition, our Decoder-Only LVSM achieves better performance than our Encoder-Decoder and it adopts a single-stream transformer to directly convent the input multi-view tokens into target view tokens, treating the view synthesis like a sequence-to-sequence translation task, which is fundamentally different from previous work, including SRT.

   [1]Light field networks: Neural scene representations with single-evaluation rendering

2. More Analysis regarding why we have better performance

   (1) Our Encoder-Decoder LVSM is similar to SRT (Sajjadi et al., 2022) at a high level –both use an encoder to transform input images into a set of 1D tokens serving as a latent representation of the 3D scene, which is then decoded to render novel views. However, Encoder-Decoder LVSM introduces a highly different architecture that significantly improves performance. We appreciate your suggestion of making a more thorough analysis in the main paper. We have revised our main paper and provided related ablation study experiments with a more detailed analysis (L442-L470, and Table 3). We also summarized the key differences as follows:

   • (a). Simpler and more effective tokenizer: We used a simpler and more effective patch-based input image tokenizer, which improves the performance and also makes the training more stable.
   • (b). Progressive compression for fixed-length latent tokens: Our encoder progressively compresses the information from posed input images into a fixed-length set of 1D latent tokens. This design ensures a consistent rendering speed, regardless of the number of input images, as shown in Fig. 6. In contrast, SRT’s latent token size grows linearly with the number of input views, resulting in decreased rendering efficiency.
   • (c). Joint updating of the latent and target patch tokens: The decoder of our encoder-decoder model utilizes pure (bidirectional) self-attention, which enables i) latent tokens to be updated across different transformer layers, which also means the parameters of the decoder are a part of the scene representation; ii) output patch pixels can also attend to other patches for joint updates, ensuring the global awareness of the rendered target image. Prior work SRT (Sajjadi et al., 2022) has a cross-attention-based design for its decoder, which doesn’t support these functions. We experiment with an LVSM variant by adopting SRT’s decoder designs, which leads to significant performance degradation, as shown in Table 3 of the revised paper (this single change will make PSNR will decrease ~3.5dB ). We have also ablated each of the above functions individually, showing their effectiveness, and discussed this in detail from L458-469.

   (2) Our Decoder-Only LVSM further pushes the boundaries of eliminating the inductive bias and bypasses any intermediate representations. It adopts a single-stream transformer to directly convent the input multi-view tokens into target view tokens, treating the view synthesis like a sequence-to-sequence translation task, which is fundamentally different from previous work, including SRT. We will discuss why decoder-only has better performance in point 3 of this reply.

   (3) In addition, we have implemented many other design choices to make our scalable and effective training happen, including using plucker rays to better represent the camera information, predicting patch pixels instead of ray pixels for more efficient rendering, integrating Flash-Attention, Gradient Checkpointing, and mixed precision training, for more efficient training, using QK-norm for more stable training, etc. These technical designs are important for final high-quality results.

Unfinished. Please keep reading the comments.

3. Explanation for the Different observations of ‘decoder-only’ architecture

   The superior performance of our decoder-only model compared to the encoder-decoder model aligns with our observation in Table 2 of the main paper: reducing the number of encoder layers while increasing decoder layers improves the performance of the Encoder-Decoder LVSM.

   This contradictes with SRT’s observation simply because LVSM and SRT have highly different decoder architectures,(which again proves LVSM is not a reimplementation of SRT). To illustrate, we use a (bidirectional) self-attention block for each decoder layer while SRT uses pure cross-attention. (Further details are discussed in point 2 of this reply.) Our decoder block design achieves superior performance, which makes the decoder-only achieve the best results according to Table 1.

4. LVSM Performance Under Unknown Pose Settings

   While we did not explicitly train LVSM under unknown pose settings as done by SRT, the model demonstrates promising single-view synthesis capabilities when trained solely on multi-view data, as evidenced by the video results available on our anonymous webpage. This suggests that LVSM has the potential to handle tasks without explicit pose information, as single-view synthesis inherently falls into this category—since a single input view can correspond to an arbitrary pose.

   We believe that training LVSM specifically on unposed/single-image inputs could further enhance its performance in such scenarios. However, due to time constraints, we were unable to explore these experiments at this time, leaving it as a promising future work.

5. Assessing the geometry accuracy

   We sincerely appreciate your suggestions for assessing geometrical accuracy. Below is our detailed response:

   • Our model can be used to reconstruct 3D mesh from sparse views. In the “Results on Cross-Domain Datasets” section of the updated anonymous webpage, we show our methods can render consistent view synthesis results from cross-domain object-level datasets (such as the hotdog example from the NeRF synthetic dataset). We can then reconstruct a 3D mesh from the view synthesis video using 2DGS(same as latentSplat you mentioned), the result can be seen here: https://3dviewer.net/#model=https://lvsm-web.github.io/mesh/hotdog.ply . We didn’t show mesh reconstruction results in the paper since we focused on view synthesis. However, we are happy to include more results in our final revised paper if you think this is important.
   • For the single-view case, we want to emphasize that single-view synthesis is not the primary focus of this project. We include some single-view synthesis results solely to demonstrate that our model has a certain level of 3D understanding, such as depth understanding, rather than focusing on pixel-level view interpolation. We will clarify this point more explicitly in the final revised version of the paper. Furthermore, we believe that single-view synthesis inherently involves scene scale ambiguity, making fair quantitative comparisons and error map visualization particularly challenging. The MVSplat work you mentioned shows an error map under two input view settings, which don’t involve scene scale ambiguity

Unfinished. Please keep reading the comments.