Cameras as Rays: Pose Estimation via Ray Diffusion

According to Eq.7, the patches of all available images are jointly processed, which is computationally expensive. As reported by the authors, training the diffusion model takes four days on 8 A6000 GPUs, which is much slower than RelPose and RelPose++

The per-iteration training speed of our method is similar to that of RelPose++, but we train for half the iterations, so our total training time is about half that of RelPose++ (4 days vs 8 days of training on 8 A6000s). As we report in Table 6, the inference speed of our Ray Diffusion method is 2.6X faster than RelPose while our Ray Regression method (which requires a single forward pass) is 220X faster.

the explanation of Eq.5 is a bit confusing. What is the “identity” camera? Are there any constraints on this equation? Does it still hold when the image depicts multiple planes?

By identity camera, we refer to a camera for which both the intrinsics and rotation are identity matrices: $K=I, R=I$. We have now clarified this in the text.

If we have a camera with rotation $R$ and intrinsics $K$, the direction of the ray corresponding to a (homogenous) pixel coordinate $u_i \in \mathbf{P}^2$ can be computed as $R^T K^{-1} u_i$. Given predicted directions $d_i$, we thus need to compute optimal $R, K$ such that $R^T K^{-1} u_i = d_i$, or equivalently, $(K R) d_i = P d_i = u_i$. Note that this is an ‘equality’ in homogenous representations (up-to-scale). Solving this corresponds to finding the optimal homography matrix such that $H d_i = u_i$.

Our terminology stems from an alternate interpretation where ‘directions’ can be thought of as points on the plane at infinity (in projective 3D space) i.e. a direction $d_i \in \mathbf{P}^2$ implies a point $(d_i,0) \in \mathbf{P}^3$. In this interpretation, finding $K,R$ is equivalent to finding the homography that relates the images of this plane at infinity under the two cameras: identity camera $(K=I, R=I)$ and the regular camera $(K=K, R=R)$ (see Sec 8.5 in Hartley & Zisserman).

Please note that this does not have anything to do with planar surfaces present in the scene; and eq 5 holds for any scene geometry.

As shown in Table 1, sometimes, the performance of the presented method decreases when more images are involved.

This issue appears to be due to a bug when computing our camera intrinsics at training time (see top-level comment). Now that the bug has been fixed, the performance does not decrease with more images as expected, as shown in Table 1. We apologize for the confusion.

As reported in Table 1, it seems that the presented over parameterization method plays a crucial role in the framework. The performance of Ray Regression (Ours) surpasses that of R+T Regression by a considerable margin. The diffusion model only results in a 3.8% improvement in the case of two images. To my understanding, the superior performance is primarily attributed to the least-squares optimization which accounts for a robust estimation. However, it is still quite confusing why the pose estimation benefits from the ray representation.

We agree with the reviewer’s assessment that over-parameterization plays a crucial role, but the other important source of improvement is the tight coupling between the ray estimation and local features. We respectfully disagree with the reviewer’s statement that performance is “primarily attributed to the least-squares optimization which accounts for a robust estimation.” To verify this, we conduct an experiment that still makes use of our patch-predicted rays but reduces the “robustness” of the least squares optimization. Specifically, we use a randomly sampled subset of the 256 predicted rays per camera to recover the camera extrinsics. We compute the rotation accuracy for our Ray Regression model on unseen object categories for N=8 images:

As we can see, accuracy goes up very quickly, even with rather few rays. Of course, the least-squares optimization gives robustness (e.g. comparing 6 rays vs 16 rays), but we believe the primary gain is from the accurately estimated rays by the local-level joint reasoning of our network.

Basically, the idea is to regress a ray represented as a 6D vector for each patch in the RGB image. It is arguably more challenging than predicting R and T. The difficulty lies in two aspects. First, it is a dense prediction problem. Second, it regresses 3D information from RGB images. One could also predict the corresponding 2D coordinates in the right image for each patch in the left image as an alternative. Intuitively, it is easier to predict 2D coordinates than 6D ray vectors. The authors argue that such a method could struggle in sparse view settings due to insufficient image overlap to find correspondences. It is unclear why the presented method is able to achieve better robustness.

We agree that predicting dense 2D correspondences between image pairs is an interesting alternative. However, converting such 2D correspondences to 3D cameras (in the presence of uncertainty, partial visibility, etc) is still challenging. Moreover, this is fundamentally a pairwise prediction task and is hard to extend to multiple images where the question of which images should be matched is also uncertain. Because our solution is not restricted to such pairwise reasoning, it bypasses this issue altogether. In addition, the representation inferred is directly convertible to global cameras.

We also note that our COLMAP baseline essentially does the sparser version of such a 2D correspondence-to-camera pipeline using a state-of-the-art feature matcher. Such methods lack robustness because of insufficient 2D correspondences.

Moreover, it is confusing why the presented method can recover the translation. According to Eq.3, m is coupled with the translation t. Predicting m then demands a requirement of capturing information about the camera translation. However, the actual input of the network is a cropped image. The information regarding t loses after the cropping.

Thank you for pointing this out. We do provide the bounding box information of the crop to the network which is important for reasoning about translation (as the reviewer notes). Specifically, we concatenate the pixel coordinate of each patch (in normalized device coordinates with respect to the uncropped image) to the spatial features for both the ray regression and ray diffusion models that we train.

We apologize for missing this detail in the earlier version and have now clarified this in the text and added the pixel coordinates to Eqs 7 and 11.

I sincerely thank the authors for the rebuttal. Some of my concerns have been addressed, but my major concern still exists. The advantages compared to correspondence-based methods are still unclear to me. The authors argued that these methods need pairwise images as input while the presented method predicts the camera pose from the single-view image. However, to estimate the camera pose, the most straightforward idea is to somehow predict the pose from a single image. Since such a single-view prediction doesn't work well, pairwise images are utilized to make the problem easier. In the presented method, the predicted $\mathbf{d}$ represents a direction in the canonical coordinate system, which means the network has to learn the camera rotation from the current view to the canonical view. Moreover, as I mentioned before, the authors propose to learn the direction for each image patch. Intuitively, such a dense prediction from a single view is more challenging. I don't understand why the method achieves promising results while the correspondence-based approach fails.

As updated by the authors, ray diffusion works worse than ray regression in camera rotation estimation, which weakens the claimed contribution. More importantly, the authors have found that there is a strong bias in the CO3D dataset and even predicting a constant rotation works better than RelPose++. This makes the conducted experiments less convincing. Given that the presented method predicts directions in the canonical frame, there is a potential risk of it being overfitted to this observed bias. The authors added some qualitative results on MegaDepth, which are not supportive enough.

I disagree with the authors that the comments on the experiments are unreasonable and counterproductive!

First, I don't think the learning-based camera pose estimation is an "emerging" research area. Even for the object-centric scenarios, some studies, such as [A, B, C], conducted experiments on other datasets. The authors argued that their work clearly represents a step forward for the area, but this is overclaimed to me with the experiments conducted on a single dataset.

Second, the authors have already reported some qualitative results on MegaDepth, which means they managed to test the method on MegaDepth's benchmark. The quantitative results are still missing, so I have a reason to wonder if the method can really work on the scene reconstruction dataset. Some studies like [D] reported experimental results on several datasets, using pairwise images as input. The authors asserted the advantages of their method compared to these competitors, but none of those datasets are examined in this paper.

I agree with the authors that these additional experiments cannot be done overnight, so I keep my previous score based on the quality of the current submission. Again, I don't see a point that the rating is unreasonable.

[A] Zhou, Xingyi, et al. "Starmap for category-agnostic keypoint and viewpoint estimation." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

[B] Xiao, Yang, et al. "Pose from shape: Deep pose estimation for arbitrary 3d objects." arXiv preprint arXiv:1906.05105 (2019).

[C] Ahmadyan, Adel, et al. "Objectron: A large scale dataset of object-centric videos in the wild with pose annotations." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[D] Sun, Jiaming, et al. "LoFTR: Detector-free local feature matching with transformers." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

This tight cropping ensures that most rays sampled pass through this common object in all views, which could provide added benefit to the propose approach. The cropping also provides a disadvantage to feature matching approaches such as used by COLMAP.

We would like to clarify that for the COLMAP baseline, we use the entire image, and for the PoseDiffusion baseline, we center-crop the image (following their protocol). The reviewer is correct that our training with cropped images could provide additional contextual information, but we based our experimental setup on RelPose.

The metrics only measure if the camera rotation is within 15 degrees of correct angle and within 10% of the scene scale in position.

In Tables 10 and 11, we analyze the rotation and camera center accuracies at a variety of thresholds. We have also added a new measure of rotation and camera center using AUC which evaluates accuracies at all thresholds in Tables 8 and 9, and visualize the threshold vs accuracy curve in Figure 11.

one concern I have is that the authors often say "sparse-view" when it would be more accurate to say "wide-baseline"

We thank the reviewer for bringing up this discussion. We agree that “sparse-view” may not be the optimal term. However, even the term “wide-baseline” may not apply in all scenarios, e.g. 2 cameras that are spatially close but viewing different directions. Perhaps a more technically accurate term would be “sparsely sampled views,” e.g., if we think of a lightfield $L(x,y,u,v)$, sparse vs densely sampled views differ in the density of the sampling of $L$. We would be happy to emphasize this in the text, but if the reviewer does not have any objections, we would prefer to stick with the current title to follow convention in prior work (e.g. sparse-view reconstruction).

Also, I found the camera visualization in Figure 5-9 to be confusing. Without the context of the 3D object or the coordinate system and with only a few cameras, it's hard to interpret what I'm looking at. In many cases it's not even clear which cameras belong to which algorithm's results.

We have re-made all the qualitative results figures starting from Figure 5. We respectfully ask the reviewer for feedback on the new presentation of results.

I'm curious whether the authors think that this method would be effective in more realistic multi-view imaging environments where the imagery is not tightly cropped to just one object and where the camera motion could be more general?

Our method can generalize to such setups. Even though our method was trained with tight crops on object-centric CO3D, we find that it generalizes zero-shot to RealEstate10K even though we used center crops (following PoseDiffusion) and the dataset has different camera trajectories. Similarly, our MegaDepth experiments use center cropping and the dataset has dramatically different camera motion.

Have any experiments been run to see if this method works on images "in the wild"?

We include results on self-captured data in Figure 6. We found that our method effectively generalizes to object-centric captures of object categories that are not found in CO3D.

In the the experiments, while is COLMAP used with SuperPoint features and SuperGlue matching? No justification is given. Is this expected to perform better or worse than vanilla COLMAP on this dataset?

In our early experiments, we found that COLMAP with SuperPoint features and SuperGlue matching performs better than COLMAP with SIFT features and nearest neighbors matching on CO3D.

The authors announce that the traditional representation of pose maybe suboptimal in neural learning in the part of introduction. However, no further discussion is given. More specific explanation is necessary, and the comparison with the proposed novel representation of pose is also required.

We apologize if this was unclear, but by “traditional representation of pose” in neural learning, we meant global parameterizations of 6D pose in the form of rotation and translation. Our experiments do highlight the benefits of our approach which represents cameras as rays compared to representative approaches which represent cameras using the “traditional representation.” In particular, in Table 1, we demonstrate that our representation improves rotation accuracy by 58% for regression-based approaches (Ray Regression vs R+T Regression) and 22% for diffusion-based approaches (Ray Diffusion vs PoseDiffusion w/o GGS). The improvement of our representation over comparable methods that use the traditional global representation of cameras holds for camera center accuracies and all numbers of images.

We hypothesize that this improvement is due to the following factors:

1. Predicting bundles of rays is particularly well-suited to transformer-based set-to-set inference.
2. Local features enable reasoning about low-level features like correspondences that are not possible in existing global-feature parameterizations in prior work.

The authors fail to state more details of the proposed network architecture. Moreover the training detail is also required.

We use standard network architectures: DINOv2 for feature extraction and DiT for regressing and diffusing camera rays. We have included additional training details such as dataset preparation, model architecture, and diffusion hyperparameters in Section 3.4 Implementation Details. We would be happy to clarify any other details, and we will be releasing code to ensure reproducibility.

To demonstrate the performance of the proposed novel representation, can authors undertake more experiments on more datasets?

We have added experiments on zero-shot generalization to RealEstate10K (following PoseDiffusion) as well as newly trained results on MegaDepth (see top-level comment). We find that our method has significantly better zero-shot generalization compared to previous methods in this scene-centric setup.

The punctuation is necessary at the end of each equation, please check it carefully. Please check the format of REFERENCES.

Thank you for these suggestions. We have added punctuation to the end of equations and updated the reference format.

It would be nice to see a "memory" requirement to run these models. Processing N image features together, I am assuming requires a good amount of GPU memory.

We thank the reviewer for this suggestion. We have added a GPU memory analysis for our Ray Diffusion model to the paper in Table 7. According to nvidia-smi, backward diffusion using our ray diffusion model on 8 images reaches a peak GPU memory usage of 3095 MiB of VRAM, which should fit on any standard GPU.

It would also be nice to see accuracy at different thresholds i.e. @5, @10, @15.

In Tables 10 and 11, we analyze the rotation and camera center accuracies at a variety of thresholds. We have also added a new measure of rotation and camera center using AUC which evaluates accuracies at all thresholds in Tables 8 and 9, and visualize the threshold vs accuracy curve in Figure 11.

It would also be nice to see an ablation study where we do not scale the poses. Most of the applications require properly "scaled" poses.

We would like the reviewer to clarify what is meant by not scaling poses. We are interpreting the reviewer’s question as whether it’s possible to not normalize camera scale at training time and instead learn metric scale. In our camera normalization procedure, the sampled minibatches of cameras are re-scaled such that the first camera always has a unit translation. This procedure is done because the ground truth camera poses are acquired using COLMAP which can produce scenes in an arbitrary scene scale, which is not consistent between sequences. Thus, it is not possible to learn metric scale since the magnitude of translations would be ill-defined. If we have incorrectly interpreted the reviewer’s statement, we politely request clarification.

The language is clear but I think the paper presentation is poor. Here are a few suggestions to improve the readability of the paper.

We thank the reviewer for the feedback and have made revisions accordingly. We have re-made all the qualitative results figures (Figure 5 onward). We respectfully ask the reviewer for feedback on the new presentation of results. For Figure 2 specifically, we are trying to convey 1) that the camera and ray representation are easily interchangeable and 2) that even noisy rays (e.g. if they do not intersect at a single point) can still be converted into a valid camera. We would appreciate any feedback from the reviewer on how to make this more clear. We have updated the caption to add more context.

I see runtime in the appendix, but how does this method scale with adding a number of views? Instead of 8 what if I had 32? What are the memory requirements and inference runtime graphs?

Although our model is trained with 8 images, we find that it can generalize effectively to more images by re-sampling mini-batches during the backward diffusion process. Specifically, at each iteration of DDPM, we can sample batches of 8 images (keeping the first image fixed), and predict the $X_0$s for each batch separately. With more images, the memory cost is constant but the runtime scales linearly. We find that our performance remains consistent with more images despite training with only 8 images, as we report below and added to Table 5 in the paper

What are your views of the applicability of this method for "scene" centric datasets?

Please refer to the top-level response. We have added experiments on zero-shot generalization to RealEstate10K (following PoseDiffusion) as well as newly trained results on MegaDepth.

Authors say they stopped the backward diffusion process early and found that those estimates were better. How early did they stop the diffusion? And how did they choose when to stop? It would be nice to see some ablation analysis.

In our initial experiments, we found that our accuracy metrics generally peak around T=30 (where T=100 is complete noise and T=0 is the “normal” final diffusion timestep). We have added an ablation that shows performance at all timesteps in Figure 12.

I found the camera visualization in Figure 5-9 to be confusing.

We have re-made all the qualitative results figures (Figure 5 onward). We respectfully ask the reviewer for feedback on the new presentation of results.