Cameras as Relative Positional Encoding

We thank Reviewer BUBt02 for their thorough review, and recognizing strengths of the work: clarity, soundness of experiments, scope and contribution, and simple implementation.

Q: "How much does including intrinsics help over just extrinsics (GTA)?"

We ran additional experiments to quantify this, in particular by evaluating GTA+CamRay to measure the improvement from including intrinsics information. We observe that intrinsics are generally critical: GTA+CamRay outperforms GTA significantly. Importantly, we also see that PRoPE and PRoPE+CamRay both outperform GTA+CamRay. This shows that the method used to condition on intrinsics matters; the PRoPE-based approach for this conditioning is empirically advantageous than raymap based such as Plucker. Tables can be found below.

Q: "Is your method limited to just the pinhole camera?"

We have not yet evaluated PRoPE on distorted images; but we believe our main conclusions will still be highly relevant hold. Potential approaches for handling distorted cameras are discussed in our response to Reviewer GRaq; we view these as promising directions for follow-up work. We will update our paper with discussion.

---

More Analysis: LVSM for Novel-View-Synthesis

Here we report the complete analysis of different conditioning approaches with LVSM on Re10K dataset. The setting is same as the the experiment reported in supplemental document Table 1 (Intrinsic-aware experiment with random 1-3x zoom-in in during training) . Here we additionally report results with GTA+CamRay, as well as the evaluation on OOD intrinsics with 1x -7x zoom-in in at test time.

Analysis

• GTA v.s. GTA+CamRay: GTA itself does not encode camera intrinsics. While the network can implicit learn the intrinsic from pixels, it is much less effective than explicitly providing the intrinsic to the model.
• Plücker v.s. GTA+CamRay: Both of them provide {extrinsic, intrinsic} to the model. GTA+CamRay consistently outperform Plücker by ~2db on PSNR indicates attention-level conditioning is superior than pixel-level conditioning.
• GTA v.s. PRoPE: A simple change of introducing intrinsic into the formula (one line change) boost performance by 4db-6db on PSNR.
• GTA+CamRay v.s. PRoPE. As camera intrinsics gradually becomes OOD, the margin between attention level conditioning (PRoPE) and pixel-aligned conditioning (GTA+CamRay) increases for 0db to 0.7db. This indicates attention level conditioning is more robust to OOD data.
• PRoPE v.s. PRoPE+CamRay. On top of attention-level conditioning, which is applied per-token, CamRay additionally provides pixel-level conditioning thus could further help with the performance. And the fact that it only affect performance marginally indicates the effectiveness of PRoPE on encoding intrinsics.

Metric: PSNR (higher is better)

Metric: SSIM (higher is better)

Metric: LPIPS (lower is better)

---

More Analysis: UniMatch for Stereo Depth Estimation.

Here we report the complete analysis of different conditioning approaches with UniMatch for stereo depth estimation. The setting is same as the the experiment reported in main paper Table 2, but due to resource constrain, these models are trained with 1/8 of the resources (2 GPUs x 50k steps) comparing to what we reported in the paper (8 GPUs x 100k steps). Here we report the average relative error reduction (more negative is better) across three test sets—RGBD, SUN3D, and Scenes11—under test-time zoom-in of {1.0x, 1.5x, 2.0x, 3.0x} to simulate cameras with OOD intrinsics.

Analysis. Across all metrics and zoom-in levels, PRoPE consistently outperforms GTA and GTA+CamRay with noticeable margins, highlighting the effectiveness of encoding camera intrinsics directly at the attention level. Notably, under the most extreme OOD condition (3.0x zoom), PRoPE continues to yield substantial gains (22–38%), whereas GTA+CamRay’s improvements drop to just 4–13%. This demonstrates the superior generalization and robustness of PRoPE's design in challenging OOD scenarios. When using CamRay with GTA, it consistently improves the performance by 1%-4%. We note that CamRay does not always bring benefits to PRoPE: this indicates that PRoPE alone is already conditioning effectively on intrinsics.

Metric Improvement: abs_rel (more negative is better)

Metric Improvement: sq_rel (more negative is better)

Metric Improvement: rmse (more negative is better)

Metric Improvement: rmse_log (more negative is better)

We thank Reviewer 19jr for their comprehensive and constructive feedback. We include responses to key points below, which we will use to strengthen our paper.

"Would GTA+CamRay or CAPE+CamRay close the gap to PRoPE?"

This is an excellent question. We have run a more exhaustive set of experiments to answer it; to reduce redundancy, tables can be found in our response Reviewer BUBt.

Importantly, we observe that PRoPE and PRoPE+CamRay outperform GTA+CamRay for both novel view synthesis and for stereo depth estimation, despite the fact that these methods include the same information and invariance properties. This corroborates our finding that PRoPE's attention-level intrinsic conditioning is important. We will include these results in our updated paper.

"Can you ablate Q/K vs. Q/K/V/O injection?"

We have run additional experiments ablating this decision on LVSM with the Re10k dataset (with intrinsics augmentation during training). Overall, they are consistent with the results in the original GTA paper: Q/K/V/O injection produces small but consistent improvement in results..

Metric: PSNR (higher is better)

Metric: SSIM (higher is better)

Metric: LPIPS (lower is better)

Q: "Have you tested PRoPE under more challenging camera models, such as fisheye lenses or other camera model?", "How numerically stable is PRoPE when projection matrices are ill-conditioned or when intrinsic parameters vary significantly?"

We have not yet evaluated PRoPE for heavily distorted images, but we believe the key conclusions in our paper still apply to these cases. We discuss potential approaches for handling distorted cameras in our response to Reviewer GRaq. This would be an exciting direction for followup work. Degenerate cases: we have not observed any instability associated with ill-conditioned projection matrices, but we agree that this is a possible in adversarial data. We will emphasize it more heavily in our future work section. We have not observed any instability associated with significantly varying intrinsics, which we do evaluate it via OOD intrinsics augmentation (up to 7x zoom-in).

"Conceptual contribution is relatively modest"

We agree with the reviewer that PRoPE builds heavily on existing ideas in relative positional encoding. We also believe, however, that the practical significance is still large. Plücker raymaps remain the dominant approach for conditioning multiview transformers, primarily because they unify representation for both camera intrinsics and extrinsics. Our work aims to show that there are better ways to implement these models, which is increasingly important as they are scaled. We also believe that the technical survey and comparative analysis in the paper will be a valuable resource for the 3D computer vision community.

Dear Reviewer,

We would like to thank you for your thoughtful follow-up.

For you concern on Would GTA+CamRay or CAPE+CamRay close the gap to PRoPE?, we would like to kindly point out the following observations from our experiments:

1. PRoPE is much more robust than GTA+CamRay to OOD intrinsics. In both the LVSM and UniMatch experiments, the gap between the two gets larger and larger when the test data contains more and more OOD intrinsics:

• LVSM: From 1x -> 7x, PSNR gap is 0.0 -> 0.7; LPIPS gap is -0.009 -> 0.044
• UniMatch: From 1x -> 3x, RMSE improvement gap is 8% -> 16% (similar for all other metrics)

2. PRoPE+CamRay consistently outperforms GTA+CamRay in all experiments : This suggests PRoPE + CamRay should be the go-to choice over GTA+CamRay.

3. GTA+CamRay itself is novel -- An outcome of our thorough analysis. The thorough analysis of different approaches is our major contribution, which leads to the both the GTA+CamRay and PRoPE. It is worth to note that none of the existing works realize this solution even though it is simple. We believe our analysis [not only PRoPE] would bring valuable insights to the community.

We appreciate reviewer again for the valuable feedbacks. These results and analysis will be added into the revised version.

We thank Reviewer GRaq for the thoroughness of their review, including: recognizing the soundness of proposed solutions, ability to retain invariance with CamRay, and value of the spatial cognition experiment.

Q: "Also report CAT3D+GTA and UniMatch+GTA", "more comprehensive set of experiments to clearly demonstrate the benefits of [PRoPE]".

We thank the reviewer for these suggestions: we agree that more comprehensive experiments would strengthen our conclusions. We have therefore run more experiments using a more thorough combination of camera conditioning techniques on both UniMatch for stereo depth estimation and LVSM for novel view synthesis, both with augmented intrinsics in training to highlight the difference benefits of PRoPE. To minimize redundancy, we include these results in our response to Reviewer BUBt.

The new results corroborate the benefits of PRoPE that were originally shown in Figure 3b and Supplemental Table 1 in our submission. We find that PRoPE consistently outperforms GTA. Importantly, both PRoPE and PRoPE+CamRay outperform GTA+CamRay, despite the fact that these methods include the same information and are all invariant to the choice of global frame. This supports our finding that attention-level intrinsic conditioning (as in PRoPE) is important.

While there is not enough time to re-run the CAT3D experiments within the rebuttal period (as we don't have access to its code and need assistants from its authors), we hope that these additional results help alleviate your valid concerns. We will include both the results and additional analysis in the updated version of our manuscript.

Q: "Could your model be easily extended to account for distortion?"

This is an excellent point; we will update our paper with discussion about this limitation and possible future directions. We believe that our high-level conclusion that relative information is important still applies. As the reviewer notes, the most direct route for incorporating distortion into our method is to undistort images before passing them into the model. Other options include injecting projective approximations (e.g., via linearization) into the attention-level encoding (e.g., each token/patch have a different intrinsic K), while relying on CamRay for more fine-grained distortion handling. It would be interesting to explore these methods for datasets with known distortion parameters. We consider this out of scope for our work, but agree that it is an exciting area for future research.

Q: "Can you quantify [...] on different magnitude of viewpoint changes?"

We use the standard test sets in each task which contains both small or large viewpoint changes. However it's challenging to quantify based on viewpoint magnitude although that is a good suggestion because both rotation, translation, and perspective changes need to be considered. And to our best knowledge there is no standard way of doing so.

Q: "Have you considered encoding epipolar information between pairs of images?"

Good question. In our initial experiments, we explored encoding the relative geometric distance between ray pairs (from corresponding pixels) in 3D space using the fundamental matrix. Under this approach, rays lying on the same epipolar plane would have zero relative distance. However, we found this encoding to be fundamentally ambiguous -- multiple distinct ray configurations can yield identical relative distance values. This limitation means that epipolar constraints alone cannot capture the complete relative geometric information needed for effective intrinsic encoding, resulting in undesired performance.

We thank the reviewer for recognizing the strengths of our work in terms of method, writing, and the consistency of performance improvements. We also appreciate the constructive feedback, which we respond to below.

Q: "Unclear intuition for CamRay. [...] Additional explanations would be helpful."

We will update our paper to include more discussion on CamRay.

The main takeaway of CamRay is that token-level and attention-level camera encodings can be complementary. While PRoPE incorporates camera intrinsics through relative positional encoding at the attention-level, it treats each patch (token) as a unit and does not explicitly distinguish between individual pixels within a token. CamRay, on the other hand, enriches input patches with pixel-wise ray information. This adds another layer of positional detail on top of what PRoPE offers, which we empirically find improve results.

We will also include more analysis, which we discuss next.

Q: "Limited evaluation of CamRay", "Limited systematic analysis"

We agree that a more exhaustive set of experiments would strengthen our paper. We have run more experiments, which include: more combinations of camera conditioning techniques, for both in-distribution and out-of-distribution data, on both the NVS and depth estimation tasks. Unfortunately, time constraints prevent us from sharing results for the CAT3D or spatial cognition tasks in time for the rebuttal deadline.

Result tables can be found in our response to Reviewer BUBt. We will update our paper with these new results, which corroborate the key takeaways from our initial submission: relative encoding is critical for performance, and including intrinsics information via PRoPE provides further gains. We also find that CamRay is useful as a tool for analysis. GTA+CamRay and PRoPE+CamRay both contain the same information and are both invariant to the world coordinate system. However, PRoPE+CamRay consistently results in better model performance. This can be interpreted as an ablation that validates the usefulness of PRoPE's intrinsics-aware relative encoding.

Q: Do varying focal length experiments "consider context views with varying focal length, or rendering target views at different target focal lengths, or both?"

Both context and target views have augmented focal lengths. We will clarify this in the writing.

Q: "Why [relative encodings] should generalize better to unseen relative pose configurations"?

Thank you for pointing this out. We agree with your statement: "PRoPE should be able to generalize to unseen absolute poses, as long as the relative poses have been seen during training". The experiments we conducted would be better described as testing "generalization to unseen number of extrinsics/cameras" (the analogy to RoPE generalizes better to longer sequence in LLMs). This affects the batch-level statistics* of the extrinsics but not the extrinsics themselves. We will revisit Figure 1b and our writing to make sure it is more precise.

*Consider: for each camera, the distance to the closest other camera. With the same scene but more cameras, this distance will usually decrease.