GIM: Learning Generalizable Image Matcher From Internet Videos

Weakness 1: Thanks for your insightful comment. Though naively adding random internet videos can potentially introduce noise or bias, we have demonstrated in the experiments that by selecting tourism videos captured at diverse geo-locations and time, GIM can effectively generalize to various zero-shot inputs, e.g., images from different domains and BEV point clouds. As we discuss in the response to all reviewers, we have trained another model with 100h of data showing that GIM scales well with more video data.

In terms of other diverse datasets, we focus on internet videos since they are arguably the most accessible and scalable data source. This ensures that the performance improvement of GIM can be reproduced by almost anyone on any future architecture. However, the key to the generalization of GIM is the data diversity and scale, rather than the source of the videos. Hence, we expect that as long as the dataset is diverse and large, it will work with GIM. So for example, adding proprietary videos captured in various environments, will likely further boost performance.

We will clarify the importance of our video selection strategy and the key to the generalization of GIM in the camera-ready paper.

Weakness 2: We completely agree that preventing overfitting is crucial. Specifically, overfitting can happen if the training videos are not diverse enough and biased towards certain types of scenes. An effective and implicit regularization that is applied in GIM is the selection of video data. We achieve this by selecting video data from diverse geo-locations and scene conditions; we focus on YouTube tourism videos and describe our search and filter strategies in Sec. 3 of the paper. Please refer to Table 6 of the paper for detailed statistics of the selected videos. Another potential factor in preventing overfitting is the amount of data. The continually growing amount and diversity of internet videos effectively reduces the risk of overfitting. We also added strong data augmentations during GIM training to make the model resistance to noise. Our experimental results in the zero-shot evaluation benchmark (ZEB) demonstrate the effectiveness of this implicit regularization approach.

Finally, though never used in our video selection process, evaluating the trained model on the ZEB benchmark can also verify whether a set of candidate videos are potentially biased or noisy.

We will include the above answers in camera-ready paper and hope they sufficiently address your concerns.

Weakness 3: For the selection of internet videos, our recommendation is to choose videos with high resolution, clear images, long duration and small number of scene transitions. To ensure these points, we choose tourism videos. They are generally 0.5-2 hours long, with a person holding the camera and recording his/her travel, mostly without transitions. GIM also has some robustness to transitions in the video since drastic changes from one frame to another make it very difficult for matches to survive after robust fitting. Even if a few matches survive, it is very difficult for the label propagation to continue. Finally, GIM includes strong data augmentations during training, which introduces noise to make the model more resistant to various sources of error in the input data.

We will include the above answer in camera-ready paper.

Question 1: As mentioned in the answer of weakness 1, we did not randomly download videos from the internet. Instead, as mentioned in Sec. 3 of the main paper, we selectively used tourism videos for training. The selected tourism videos are usually captured by a person traveling around various places. Even only on YouTube, we can find tourism videos spanning thousands of hours and covering various regions, years, times, weather conditions, and lighting. These videos are not limited to a single type of scene; they include diverse activities like walking, cycling, driving and more. This ensures the data diversity and minimizes the bias.

Based on your suggestion, we will describe our video selection process and its effectiveness in details in our camera-ready paper. The code will also be released upon acceptance.

Question 2: Please refer to the details we have mentioned in weakness 3 for information how we handle noise and varying quality.

Some pre-processing is indeed applied. First, we search YouTube for tourism videos using the keywords 'walk in' or 'walk through', and then downloaded videos that are at least 30 minutes long. Then we remove the first 5 minutes and the last 5 minutes of the video and keep only the middle part, because there may be a quick preview or summary which can contain non-smooth scene transitions. We will include these details in the appendix of camera-ready paper for clarity.

Question 3: Please refer to the response of weakness 2.

Thank you for the in-depth review. We will address all 4 suggestions in the camera-ready paper and provide responses to the major comments below.

Weakness 1.: This is a very good suggestion. To facilitate better zero-shot evaluation, we will include more diverse real-world indoor scenes (e.g., SUN3D, ScanNet V2) during the code release of GIM and ZEB.

Question 1.: Thank you for this insightful question. Since the ``iterative self-training process'' could be interpreted in different ways, we provide answers based on two possible interpretations. We will also clarify this in the camera-ready paper.

(Case 1) Using a fixed set of videos in multiple rounds of iterative self-training: In this case, each round of self-training uses the current best GIM model (trained in the previous round) to re-generate the labels on the same set of videos, and then updates the GIM model with the re-generated labels. The expected improvement would come from better video labels and longer training time on the same set of videos. In terms of the video labels, Table 2 of the main paper shows that a better label generation method leads to better GIM performance. Hence, we would expect that this bootstrapping could in principle provide a further performance boost. However, the improvement of a single method may not be very significant in practice, due to the use of multiple complementary methods for label generation. In terms of the longer training time, image matching models are prone to overfitting, hence early stopping is applied during training. GIM does not change the baseline architecture or training curriculum and might also exhibit the overfitting problem for long training times, if the video dataset is too small and not expanded over time. Hence, if ``iterative self-training'' refers to case 1, it may slightly improve results but will probably not be as effective as expanding the video dataset at the same time. We discuss this case below and expect it to be the best way to boost the performance of GIM with iterative self-training.

(Case 2) Expanding the number of videos in multiple rounds of iterative self-training: In this case, each round of self-training uses the current best GIM model (trained in the previous round) to generate labels on a new set of videos, and then performs the next round of self-training with both the old and new videos. Table 2 of the main paper shows that the most effective way to improve GIM is to add more video data, rather than generating better labels on a fixed set of videos. Hence, given the limited time and training resources (GPUs) and the potentially infinite amount of available internet videos, using the current best GIM model to generate labels on a new set of videos can best improve the performance. In this case, albeit less likely, overfitting may still happen when training the model for a long time. Hence, we recommend keeping early stopping active and re-initializing GIM weights for multiple rounds of self-training.

Question 2.: We expect that GIM will also work with egocentric videos since there is nothing specific to the camera perspective in the algorithm. The more important aspect is that a similar level of diversity as in the case of the tourism videos is ensured (e.g., different geo-locations, lightning conditions, etc.). Training GIM on a mixture of diverse tourism videos and ego-centric videos may also be a good choice and boost generalization performance further. It is an interesting direction for further investigation.

Thank you for the encouraging review, your appreciation of our simple yet effective method, and your valuable suggestions.

Weaknesses (Implementation details and code release): The training part mostly follows the baseline code as mentioned in the ``implementation details'' part of the paper.

To ensure reproducibility, we will provide more details such as training time on different GPUs and for different base architectures, as discussed in the response to Reviewer-sN7h. We will also release the pre-trained GIM models, the training and ZEB evaluation data, and the code for label generation and model training.

Questions (Meaning of domain specific training): In the paper, we refer to training on standard image matching datasets (such as e.g., MegaDepth, ScanNet and unlike in-the-wild data from YouTube) as domain-specific training. In the experiments, we use the officially released models, i.e., SuperGlue (OUT), LoFTR (OUT) and DKM (OUT), as domain-specific models for GIM label generation. We will clarify this point in the camera-ready paper to avoid misunderstandings.

Weakness (1): You are right, the complementary methods are introduced to increase the number of correspondences. Specifically multi-method matching is conducted only on nearby video frames ($<$80 frame interval), in which setting complementary methods are still able to provide reliable correspondences, albeit with inferior performance. The reason that complementary methods can increase the number of correspondences is that different methods often generate correspondences with different distributions. For example, RootSIFT generates correspondences on key point pixels. DKM outputs are mostly located on non-key point pixels. This strategy enhances the correspondence density between nearby video frames, which allows label propagation to produce training images with a larger pose difference.

We do not just use 1 complementary method for each baseline, but rather all available methods that are inferior to the baseline, i.e., for training GIM$\_{SuperGlue}$, we use RootSIFT+SuperGlue to generate video labels. For training GIM$\_{LoFTR}$, we use RootSIFT+SuperGlue+LoFTR and for training GIM$\_{DKM}$, we use RootSIFT+SuperGlue+LoFTR+DKM to generate video labels. More complementary methods should in principle benefit the performance of GIM. For fair comparison, we use only inferior complementary methods so that we do not use labels generated by better methods to improve the baseline.

We will clarify both points in camera-ready manuscript to avoid misunderstandings, thanks for your question.

Weakness (2): Yes, batch size affects the self-supervision performance; generally enlarging the batch size improves the performance. We follow the official implementation of the baselines to set the batch size.

In terms of the training time of GIM, if we use 8 A100 GPUs, it takes about 4, 5 and 7.5 days to train GIM$\_{SuperGlue}$, GIM$\_{LoFTR}$, and GIM$\_{DKM}$ respectively.

Yes, GIM works on GPUs with smaller GPU memory such as the RTX-3090, which is the GPU we used for initial experiments. For label generation, it takes 15 days to process 50 hours of videos on 4 RTX-3090 GPUs. For training, it takes 9.5, 12, and 19 days to train GIM$\_{SuperGlue}$, GIM$\_{LoFTR}$, and GIM$\_{DKM}$ respectively. We apply gradient accumulation when using the RTX-3090 GPU to ensure the same training batch size as on the A100 GPU.

We will clarify these points in the appendix of the camera-ready manuscript to ensure the reproducibility of GIM across various setups.