ROBUST SPARSE AND DENSE MATCHING

Thanks for your instructive suggestions on our paper. In this work, we aim to improve the generalization performance of the matching problem by collecting large-scale datasets covering optical flow, and geometry estimation datasets. We also propose a generalist model, capable of both optical flow estimation and geometry estimation. Our method achieves the best zero-shot evaluations across all datasets for matching and pose estimation tasks, strongly validating the effectiveness of our proposed method. To mitigate the degeneration of specialized datasets caused by training on data of diverse distributions, we propose a decoupled training strategy by learning matching and uncertainty estimation in a two-stage manner. Overall, we have taken into full consideration both the specificity and generality. The best zero-shot generalization performance compared with prior generalist and specialized models as well as the competitive performance compared with specialized methods can strongly demonstrate the contributions of our work. We also report the performance of optical flow following the standard setting as specialized models. The comparison is illustrated in Tab.6. We provide the updated results by adopting the tile mechanism proposed in FlowFormer. The results are shown below:

This comparison strongly validates that if training on specific datasets under the same setting as specialized models, our method can also obtain superior performance. We will provide detailed explanations to your problems below:

Network architecture:

Thanks for pointing out this problem. We provide a detailed explanation for this problem in our first official comment. The primary reasons are twofold. First of all, the iterative optimization mechanism provides an explicit and efficient approach for exploiting geometric similarity, which can be employed in geometry estimation methods for further improvement. Moreover, conducting matching at a higher resolution can empirically introduce additional improvement for downstream pose estimation as in DKM, PATS, etc. However, popular optical flow estimation works like RAFT, FlowFormer, and GMFlow end up with the refinement at a relatively low resolution(1/4 at most). To mitigate this problem, we elaborately construct the model architecture to perform refinement at 1/2 resolution. In future work, we will conduct more comprehensive experiments and utilize your mentioned models(GMFlow, PDCNet+) for generalization evaluations with our training data.

Degeneration with optical flow

Currently, widely used datasets for training optical flow, such as FlyingThings3D, FlyingChairs, Sintel, and TartanAir, are synthesized datasets with relatively small displacements. In contrast, real-world datasets for geometry estimation often exhibit substantial changes in viewpoint. The integration of a diverse set of datasets with varying distributions, as opposed to training on specific datasets tailored for specialized tasks, may introduce inevitable interference, which accounts for the degeneration. The ablation studies in Table 1 are primarily conducted to reveal this phenomenon and demonstrate the effectiveness of our proposed decoupled training strategy in mitigating this decline. A more straightforward table for downstream pose estimation is presented below:

Leveraging our decoupled training strategy efficiently enhances the performance when the optical flow dataset is used. In Tab.2, we utilize our decoupled training strategy in default to explore the effectiveness of scaling up the diversity of training for generalization performance. The uncertainty estimation module is not involved at this matching stage so only generalization evaluations on matching tasks are reported.

We greatly appreciate your immediate correspondence. We sincerely hope that in your reassessment, you can kindly take our primary contributions to the generalized matching task for both matching and pose estimation as well as our explorations to alleviate the degeneration when training on the mixed data of diverse distributions into consideration. Thanks for your instructive and academic suggestions on our work again!

Thanks for pointing out the weakness of our work and we apologize for the unclear writing in this component. We will provide more comprehensive experiments in our revised supplementary materials.

Comparison with other methods:

In our work, we utilize the shifted-window strategy to perform self-attention and cross-attention for feature enhancement. The compared methods like GMFlow and FlowFormer also adopt the high-weight parameters. It's worth mentioning that FlowFlower utilizes the Twins-L model as its backbone for feature extraction.

Warping error computation

As briefly introduced in Sec.3.2, we warp the target frame to the reference frame according to the estimated optical flow with bilinear sampling. Given the input reference and target images $I_r, I_t,$ as well as the estimated optical flow $f$ at the grid $x$, the warped reference image can be computed as:

and the warped RGB error can be computed as:

We also compute the feature error $E_{feat}$ in the same way with reference and target features $F_r, F_t$:

The final warping error is defined as the concatenation of both warped RGB error and warped feature error. We will add detailed introduction about the implementation of warping error in our revised paper.

Uncertainty map for ransac filter

The uncertainty map serves as a manifestation of the effectiveness of sparse correspondence matching. A higher degree of effectiveness in the sparse matching outcomes for a specific point is associated with an increased magnitude of its corresponding uncertainty. The cumulative uncertainties of all points within an image collectively contribute to the formation of its corresponding uncertainty map. We conduct a point selection procedure involving the uncertainty map and its corresponding matching outcomes, subsequently directing the selected points into a RANSAC filter. Our point selection process comprises two sequential steps. The initial step involves the selection of points using uncertainty as a screening criterion. The second step, based on the distribution among these selected points, constitutes a secondary selection aimed at ensuring diversity in the chosen points. In the specific context of the secondary point screening step, it entails the initial computation of Euclidean distances among the points selected during the initial screening. Euclidean distances are subsequently employed as a reference to establish weights for the secondary point selection, where greater distances correspond to augmented weights. As a preventive measure against the introduction of noise data, weights for points with distances exceeding a predefined threshold are reset to the minimum value of 1e-7.

Comparison with other optical flow methods:

Sorry for the shortage of this comparison with other optical flow methods in the main text which is placed in the supplementary materials as can be found in Tab.6. In our later research, we find that additional improvement can be obtained by adopting the tile technique mentioned in FlowFormer for alleviating the influence of position embedding at different resolutions. We provide the updated results in the following table:

This comparison strongly validates that when training on specific datasets for specialized tasks, our model still obtains outstanding performance, achieving the best results on both Sintel and KITTI datasets. We will provide comprehensive comparisons with other specialized models under the same setting of specific tasks to validate the effectiveness of the proposed method. Moreover, we will place the aforementioned comparison on optical flow estimation in our main text in the revised version.

Computation cost and inference speed

Thanks for pointing out the absence of this experiment. We select state-of-the-art methods and make comparisons with them in terms of computation costs and inference speed. The result is shown below:

We will explore to construct a more efficient model in our future work.

Thanks for pointing out the problems of our work. Here are our explanations.

Model architecture

As mentioned in the introduction, the model design is not our primary contribution in this work. We think any dense matching framework coupled with an uncertainty estimation module can serve as a generalist model. However, currently available models have their limitations. The iterative optimization mechanism provides an explicit and efficient approach for exploiting geometric similarity, which can be employed for geometry estimation. Emperiaclly, conducting matching at a higher resolution can introduce additional improvement for downstream pose estimation as in DKM, PATS, etc. However, popular optical flow estimation works like RAFT, FlowFormer, and GMFlow end up with the refinement at a relatively low resolution(1/4 at most), which is the main reason we elaborately construct the model architecture to perform refinement at 1/2 resolution.

Advantage of our organized dataset:

Thanks for pointing out this question. The re-organized of the TartanAir dataset is an important exploration in our work. Notably, the magnitude of displacement varies substantially across different datasets and scenarios. We list the displacement distribution of different datasets as well as our organized data below which can also be found in Tab.5 in our paper. It's worth noticing that our generated optical flow dataset obtains a different distribution compared to the original TartanAir dataset.

We re-organize the dataset by increasing the sampling interval to obtain a closer distribution of other real-world datasets. The computation of optical flow ground truth is illustrated in our appendix A.1. The effectiveness of our proposed dataset is obvious as shown in Tab.1 and Tab.2. We simplify Tab.2 as below to show the effectiveness of our organized TartanAir sampled(TS) dataset.

The adoption of our organized TartanAir dataset efficiently improves the generalization performance across multiple datasets, demonstrating the advantages of our organized dataset.

Ablations on multi-scale design

We are sorry for the confusion caused by the absence of this experiment. We provide the following ablation studies by inferring at previous scales for a quick validation of our multiscale design. The results are shown below and scale 3 corresponds to the refinement of 1/2 resolution utilized in our final model while scale 1 indicates the refinement of 1/8 resolution:

Furthermore, we also utilize the matching at the previous scale for pose estimation, and the performance is shown below:

The above-listed results across matching and pose estimation tasks validate the effectiveness of the multi-scale design and the necessity of applying refinement at a higher resolution. We will place this comparison in the supplementary materials and conduct more comprehensive ablation experiments in our revised paper.

Ablation Study Setting:

We apologize for the ambiguity caused by our experiment settings and paper writing. In Tab.1, we illustrate the effectiveness of the decoupled training strategy by learning matching and uncertainty in a two-stage manner. In our final experiment, we adopt this decoupled training strategy. At the matching stage, we first train the dense matching model on the mixture of datasets as shown in Tab.2 to improve the generalization performance, and the effectiveness is validated. In the second stage, we freeze the parameters of the matching model and train the uncertainty estimation model based on the warping error according to the matching model. At this stage, only Megadepth and ScanNet are utilized. The detailed training settings are mentioned in Sec.4.1.