CRAYM: Neural Field Optimization via Camera RAY Matching

Thank you for the insightful comments and suggestions, especially the series of detailed questions, which we will all answer below. We hope the answers will address your concerns.

Q: The robustness of the approach against outlier matches is not evaluated. Introducing artificial outliers (wrongly matched key points) into the dataset to assess how well the technique can handle mismatches would be of some interest. Would the matched rays' consistency and the epipolar geometry compensate for that? Or would the training diverge?

A: Yes, our Matched Ray Coherency formulation is precisely designed to compensate for wrongly-matched rays. The use of auxiliary rays serves to improve the robustness of ray matching. In practice, erroneous keypoint/ray matches do happen and they probably do not need to be "artificially introduced." For example, when the noise level increases, wrongly-matched rays will tend to happen more frequently. To this end, our robustness experiments against different noise levels (see Table 2 in the paper) can be regarded also as a robustness test on our method's ability to handle wrongly-matched rays.

Q: As stated in the literature review of this manuscript, other approaches taking advantage of the epipolar geometry and prior matching have already been designed in this context. I have difficulty understanding what is significantly different with this work apart from the sampling of additional surrounding rays and the matching ray index used to weight color prediction using image pairs. These two novelties seem rather incremental, but they nonetheless lead to strongly improved results. Question: I assume that the other keypoints-based approaches are not "self-calibrated". Is the proposed technique the first "keypoint-based" calibration-free NeRF? If it is not the case, it would be meaningful to compare against such techniques too.

A: Yes, CRAYM is the first "keypoint-based" calibration-free method, as far as we are aware of. Other approaches usually compute the reprojection distance of the matched pixels, while we compute the distance between the reprojected pixel and the epipolar line as the epipolar loss, thereby boosting the convergence of the camera poses.

Q: Adding surrounding rays around a key ray appears to be quite effective; however, the sampling of auxiliary rays is not well described in the paper. Question: How are the rays sampled?

A: Currently, we randomly sample n auxiliary rays from the k nearest rays of key rays. More details of auxiliary rays will be given in the revision.

Q: The initialization of the pose lacks details. Question: What is the effect of the pose initialization on the result?

A: As stated in the paper, we follow L2G-NeRF to perturb the ground-truth camera poses with additive noise as the initial poses. Table 2 shows the results of using different noise levels on the initial poses. In general, well-initialized poses produce robust and fast convergence.

Q: The intrinsic parameters of the camera could additionally be optimized. Question: Just out of curiosity, have you conducted such an experiment?

A: All the evaluations assume known camera intrinsics, as most of the other methods do. We have performed experiments to additionally optimize the intrinsics and they succeed in learning the intrinsics with some more interactions.

Q: In equation (4), it seems that the proposed solution considers only pairs of images. Question: How are those pairs selected?

A: We do not select image pairs for training. Instead, we randomly select a ray from the ray set generated with all the training images and their poses. If a matched ray is found for it, the matched ray pair is optimized. Otherwise, the selected ray is optimized on its own.

Q: The loss function contains many regularization factors. Question: Is the final loss hard to balance?

A: Empirically, we found that these items in the loss function promote one another rather than weakening/antagonizing one another and we do not need to tune them for an optimal balance in producing our results.

Thank you for the careful review and insightful questions. The reviewer's various suggestions are right on and we hope the rebuttal will alleviate the concerns raised.

Q: Additional results on accurate regression of camera poses?

A: Please refer to Table 9 in the supplemental material. It shows exactly the quality of pose regression by our method. The results show that CRAYM yields better pose estimation than other methods. If requested and space permitting, we can move the relevant materials to Section 4 of the main paper.

Q: It would be beneficial to showcase more experimental results on different types of datasets with an emphasis on trajectory of pose optimization and high-fidelity renderings.

A: This request coincides with what Reviewer GtQM asked above. Hence, we copy the results on the LLFF dataset below. The trajectories of LEGO from NeRF-Synthetic (Blender) are shown in the uploaded PDF file. Other trajectories will be provided in the revised paper.

Table 1: Mean scores over three metrics.

Table 2: PSNR scores over various models.

Table 3: SSIM scores over various models.

Table 4: LPIPS scores over various models.

Q: This work is implemented using grid-based representation, while the compared methods are implemented using MLP-based representation, leading to an imprecise and unfair comparison.

A: This is a fair point when considering the original implementations. However, when making comparisons to CRAYM, we did implement BARF with grid-based representations as the baseline and the comparison results are shown in paper and also in the four tables above. The gap between the BARF baseline (27.30) and CRAYM (31.60) shows that the improvement comes mainly from exploiting the matched rays in our proposed formulation. The results of another grid-based representation method, namely NeuS2, are also provided in the table at the end of this rebuttal.

Q: The comparison under sparse-view conditions lacks precision and distinction. SPARF was evaluated on the DTU dataset with only 3 views, while you used 48 images in your setting, although you have reported results for sparse input (3 views) on the LEGO data, it may not be sufficiently convincing.

A: The overall synthesis quality with sparse views is demonstrated with the whole evaluation image set. We randomly select three images from the evaluation set as the final results. The comparison with SPARF is shown below, with our method outperforming:

Q: I recommend that the authors compare with Neus2.

A: Good idea. Please first note that in Table 4 in the submitted paper, we applied a progressive feature mask on the multi-resolution hash encoding into a neural surface representation as our baseline, which combines BARF and NeuS2. The results of NeuS2, the BARF baseline, l2G-NeRF, and CRAYM are shown in the table below, with our method coming on top. Although NeuS2 produces better reconstructions than L2G-NeRF, L2G-NeRF outperforms NeuS2 in the quality of the rendered views.

Thanks for the great efforts of the authors! After carefully reading the responses both from the authors and other reviewers, most of my concerns have been addressed. I tend to improve my rating to BA now and suggest the authors add more experimental details/results and discussions in the revision.

Thank you for your care and insights in the reviews and the encouraging remarks!

Q: Reproducibility

A: All the implementation details mentioned along with other useful information will be added to the supplemental material in the revision. The source code and any data used will surely be released upon paper publication.

Q: Complexity

A: When training CRAYM, each iteration consumes around twice the amount of resources vs. other low-resource alternatives, since our method optimizes two matched rays together in each iteration. However, CRAYM converges much faster than these alternatives (10-20k iterations vs. 200k iterations) during training. During inference, the views are synthesized from the optimized field with the same rendering process and each ray is rendered separately, so the inference efficiency of CRAYM is the same as the other methods.

Q: Qualitative results, synthesized images, depth images, example key, and auxiliary point.

A: We will provide these results in the revised paper as space would allow. All other requested additions will be incorporated into the supplementary material.

Q: Is the training done only on matching key points?

A: No. Matched rays are optimized together, whereas rays without matched companions are optimized separately. Without using any matched rays, CRAYM will be downgraded to the baseline method.

Q: How sensitive is the method to the quality and density of the 2D matches?

A: In our Matched Ray Coherency formulation, we explicitly account for potentially erroneous (i.e., low-quality) 2D matches by using the matchability between two rays as a weight to either accentuate or discount the color consistency constraint; see line 66 in the paper. In terms of sensitivity with respect to the density of the 2D matches, in our experiments, we have observed that even with sparse input views and sparsely distributed matched rays, CRAYM can still notably improve the optimization convergence. We would be happy to provide such a sensitivity analysis in the revision.

We appreciate the suggestions in the review and hope the rebuttal will help address the concerns raised.

Q: Experiments were only conducted on NeRF-synthetic datasets and not on LLFF datasets. Also add Neural Image Alignment to enhance the evaluation.

A: We have evaluated our method on the LLFF dataset and the results are listed in the four tables below. The first table shows the mean scores for PSNR, SSIM, and LPIPS, comparing our method (CRAYM) to three baselines including BAA-NGP, a recent method suggested by the reviewer. Our method outperforms all the baselines over all metrics. The remaining three tables show performance numbers for each of the metrics on various models.

Evaluations on the Neural Image Alignment task will be added in the revision.

Table 1: Mean scores over three metrics.

Table 2: PSNR scores over various models.

Table 3: SSIM scores over various models.

Table 4: LPIPS scores over various models.

Q: Include more recent methods such as NoPe-NeRF and BAA-NGP.

A: The comparisons with BAA-NGP were included in the tables above, and as shown, our method outperforms all the baselines including BAA-NGP, over all three metrics PSNR, SSIM, and LPIPS, on the LLFF dataset.

As for NoPe-NeRF, we had tested the code released by the authors of that work. However, since NoPe-NeRF heavily relies on depth estimation results to adjust the camera poses, it cannot effectively generalize to other datasets, as in our setting.

Q: Figure 5 and Table 3.

A: Thanks! We will reduce the overlap in the revision.

Thank you to all the reviewers for your detailed and constructive feedback on our submission. We have addressed all the questions raised and conducted the required experiments. These primarily involved comparative analyses on the LLFF dataset, and the comparisons with BAA-NGP and NeuS2.

In our experiments, the proposed method demonstrated superior performance across various metrics compared to the alternatives. The uploaded PDF file includes figures illustrating the optimized trajectories of the different methods evaluated.

We remain open to addressing any further questions or concerns.