GPS: A Probabilistic Distributional Similarity with Gumbel Priors for Set-to-Set Matching

Thank you for the valuable comments. Below are our responses to your concerns:

1. Better Organization: Figure 2(c) demonstrates that Gumbel distributions can effectively fit KNN data, not just first-order nearest neighbors. Figure 3 presents the graphical model for computing GPS, similar to those used in probabilistic modeling like pLSA and LDA. We will restructure the paper to enhance clarity.

2. Paradigm of Applying GPS to A Specific Task: To utilize GPS, we can simply replace the set-to-set feature distances in the original approaches with our negative GPS scores and then train the networks. We will include some Python-like pseudo-code to illustrate this process.

3. More Experiments: We conducted more experiments for both point cloud matching/registration and image matching using $GPS(\alpha, \beta)$ with only 1-NN and 1 Gumbel distribution. Details are as follows:

• (a) Point Cloud Matching with PREDATOR (Huang et al. CVPR, 2021): We modified the public code from https://github.com/prs-eth/OverlapPredator by simply replacing the distances in the loss with the negative GPS scores. The results are listed below:

| Samples | 5000 | 2500 | 1000 | 500 | 250 | Ave.$\pm$Std |

| PREDATOR | 89.0 | 89.9 | 90.6 | 88.5 | 86.6 | 88.92$\pm$1.53 |

| P + GPS(1, 1) | 89.3 | 89.6 | 90.0 | 88.2 | 88.7 | 89.16$\pm$0.72 |

| P + GPS(1, 2) | 88.6 | 90.2 | 90.3 | 89.2 | 88.4 | 89.34$\pm$0.88 |

| P + GPS(2, 2) | 89.4 | 89.4 | 90.8 | 89.3 | 89.3 | 89.64$\pm$0.65 |

• (b) Image Matching with LightGlue (Lindenberger et al. CVPR, 2023): Replacing distances in the loss for SuperGlue (Sarlin et al. CVPR, 2020) with our negative GPS scores is not straightforward, because there are some numerical stability issues in the log-space. Instead, we utilized LightGlue (https://github.com/cvg/LightGlue), which is inherited from SuperGlue, for image matching. Same as we did for PREDATOR, we replaced the feature distances with our negative GPS scores for computing the loss in LightGlue. The results are listed below:

| AUC | 5 degree | 10 degree | 20 degree | Ave.$\pm$Std |

| LightGlue | 41.5 | 57.8 | 70.7 | 56.67$\pm$14.63 |

| L + GPS(1, 1) | 42.5 | 48.3 | 71.2 | 57.33$\pm$14.37 |

As we can see, in both applications, our GPS can significantly improve the baselines in terms of accuracy and robustness. Note that we have not fine-tuned our hyperparameters for GPS in the tables yet. In summary, our GPS can be applied to a wide range of applications for performance improvement.

We appreciate your comments and the insights. Before we offer a response, we were wondering whether you could clarify your comments.

Comparisons to Prior Works: You noted that “within the framework of set matching and similarity learning, many studies have introduced new methods and ideas from different perspectives.” Could you point to specific studies or methods you feel would be most relevant for comparison.

Theoretical Discussion on Assumptions: We noticed your point about the challenges of independent and identically distributed (i.i.d.) assumptions in practical applications. Could you clarify how you see this assumption impacting our methodology?

Performance Improvement Evaluation: You comment that our method showed only modest accuracy improvements. We would appreciate if you could clarify what you mean with the context of other reviews here.

We appreciate the reviewer’s comments regarding the i.i.d. assumption. The mention of i.i.d. in the paper is a side-comment meant to provide context for the use of the Gumbel distribution. It is not central to the methodology or results. The purpose of including it was to give a more complete explanation of the origins of the Gumbel distribution, but this was intended as an educational aside rather than a strict requirement of our approach.

Algorithmically, our method simply involves:

1. Computing the nearest-neighbor distances between two sets.
2. Fitting a Gumbel distribution to these distances.
3. Using the resulting parameters to measure set-to-set similarity.

The i.i.d. assumption does not constrain this process. The algorithm works effectively regardless of whether the data fully satisfies i.i.d. properties, as demonstrated by the empirical results (e.g., Figure 4). Indeed, the paper would be no different if we had omitted any reference to i.i.d., as the practical implementation is focused on fitting a Gumbel distribution to observed data.

We appreciate the reviewer’s attention to the results and would like to clarify the contributions of our work and the evaluation of its performance. Specifically:

Relevance of Datasets and Baselines: The datasets, baselines, and experimental tasks we consider are specifically chosen to align with the problem of set-to-set matching that GPS addresses. These benchmarks are widely recognized in the literature for evaluating methods in this domain, including:

1. Few-shot classification on datasets such as miniImageNet, tieredImageNet, and CIFAR-FS.
2. Point-cloud completion on ShapeNet-Part and KITTI.

State-of-the-Art Performance: GPS achieves consistent state-of-the-art (SOTA) performance across benchmarks in few-shot classification and point-cloud completion tasks (see Table 5-12).

For example, in few-shot classification: On the miniImageNet 1-shot 5-way classification task, GPS achieves an accuracy of 66.27 ± 0.37%, compared to 64.01 ± 0.32% by InfoCD (NeuRIPS 2023), 63.63 ± 0.65% by HyperCD (CVPR 2023), and 63.36 ± 0.75% by DeepEMD. This represents an improvement of up to 2.26 percentage points over prior SOTA methods. On tieredImageNet 1-shot 5-way classification, GPS achieves 73.16 ± 0.43%, outperforming HyperCD (70.58 ± 0.81%) and InfoCD (70.97 ± 0.59%) by similar margins .

CIoud completion: On the ShapeNet-Part dataset, GPS achieves the lowest L2-CD score (3.94 ± 0.003) compared to 4.01 ± 0.004 by InfoCD and 4.03 ± 0.007 by HyperCD . On the KITTI dataset, GPS achieves a Fidelity of 1.883 and an MMD of 0.302, consistently outperforming prior methods.

**Significance of Improvement - Improvements of 1-2% in these tasks are considered highly significant due to the competitive nature of existing baselines. For instance:

Novel Contributions: Beyond performance improvements, GPS introduces a novel probabilistic framework for set-to-set similarity, leveraging the Gumbel distribution to capture distributional alignment efficiently. This innovation addresses computational challenges in previous approaches while maintaining high accuracy.

Request for Clarification: We respectfully request the reviewer to clarify what is meant by "not significant" and to provide specific criteria or examples of what they would consider "more interesting results." Our evaluation rigorously benchmarks against all prior works, and GPS demonstrates clear SOTA performance.

Thanks to the author for his patient reply, which has solved most of my problems. I will improve my score. I am curious about how this method works in the field of point cloud registration. I noticed that point cloud registration is a research field with many applications. Here are some references.

RIGA: Rotation-Invariant and Globally-Aware Descriptors for Point Cloud Registration, TPAMI 2024

Rotation-Invariant Transformer for Point Cloud Matching, CVPR 2023

EGST: Enhanced geometric structure transformer for point cloud registration TVCG 2024

TEASER: Fast and Certifiable Point Cloud Registration, TRO 2021

3D Registration With Maximal Cliques, CVPR 2023

As suggested by Reviewer auE8, Reviewer S2Z9 and Reviewer YjK9, we conducted more experiments for both point cloud matching and image matching using $GPS(\alpha, \beta)$ with only 1-NN and 1 Gumbel distribution. Details are as follows:

1. Point Cloud Matching with PREDATOR (Huang et al. CVPR, 2021): We modified the public code from https://github.com/prs-eth/OverlapPredator by simply replacing the distances in the loss with the negative GPS scores. The results are listed below:

| Samples | 5000 | 2500 | 1000 | 500 | 250 | Ave.$\pm$Std |

| PREDATOR | 89.0 | 89.9 | 90.6 | 88.5 | 86.6 | 88.92$\pm$1.53 |

| P + GPS(1, 1) | 89.3 | 89.6 | 90.0 | 88.2 | 88.7 | 89.16$\pm$0.72 |

| P + GPS(1, 2) | 88.6 | 90.2 | 90.3 | 89.2 | 88.4 | 89.34$\pm$0.88 |

| P + GPS(2, 2) | 89.4 | 89.4 | 90.8 | 89.3 | 89.3 | 89.64$\pm$0.65 |

2. Image Matching with LightGlue (Lindenberger et al. CVPR, 2023): Replacing distances in the loss for SuperGlue (Sarlin et al. CVPR, 2020) with our negative GPS scores is not straightforward, because there are some numerical stability issues in the log-space. Instead, we utilized LightGlue (https://github.com/cvg/LightGlue), which is inherited from SuperGlue, for image matching. Same as we did for PREDATOR, we replaced the feature distances with our negative GPS scores for computing the loss in LightGlue. The results are listed below:

| AUC | 5 degree | 10 degree | 20 degree | Ave.$\pm$Std |

| LightGlue | 41.5 | 57.8 | 70.7 | 56.67$\pm$14.63 |

| L + GPS(1, 1) | 42.5 | 48.3 | 71.2 | 57.33$\pm$14.37 |

As we can see, in both applications, our GPS can significantly improve the baselines in terms of accuracy and robustness. Note that we have not fine-tuned our hyperparameters for GPS in the tables yet. In summary, our GPS can be applied to a wide range of applications for performance improvement.

We thank the reviewer for the valuable comments. We have conducted more experiments for both point cloud matching/registration and image matching using $GPS(\alpha, \beta)$ with only 1-NN and 1 Gumbel distribution. Details are as follows:

(1) Point Cloud Matching with PREDATOR (Huang et al. CVPR, 2021): We modified the public code from https://github.com/prs-eth/OverlapPredator by simply replacing the distances in the loss with the negative GPS scores. The results are listed below:

| Samples | 5000 | 2500 | 1000 | 500 | 250 | Ave.$\pm$Std |

| PREDATOR | 89.0 | 89.9 | 90.6 | 88.5 | 86.6 | 88.92$\pm$1.53 |

| P + GPS(1, 1) | 89.3 | 89.6 | 90.0 | 88.2 | 88.7 | 89.16$\pm$0.72 |

| P + GPS(1, 2) | 88.6 | 90.2 | 90.3 | 89.2 | 88.4 | 89.34$\pm$0.88 |

| P + GPS(2, 2) | 89.4 | 89.4 | 90.8 | 89.3 | 89.3 | 89.64$\pm$0.65 |

(2) Image Matching with LightGlue (Lindenberger et al. CVPR, 2023): Replacing distances in the loss for SuperGlue (Sarlin et al. CVPR, 2020) with our negative GPS scores is not straightforward, because there are some numerical stability issues in the log-space. Instead, we utilized LightGlue (https://github.com/cvg/LightGlue), which is inherited from SuperGlue, for image matching. Same as we did for PREDATOR, we replaced the feature distances with our negative GPS scores for computing the loss in LightGlue. The results are listed below:

| AUC | 5 degree | 10 degree | 20 degree | Ave.$\pm$Std |

| LightGlue | 41.5 | 57.8 | 70.7 | 56.67$\pm$14.63 |

| L + GPS(1, 1) | 42.5 | 48.3 | 71.2 | 57.33$\pm$14.37 |

As we can see, in both applications, our GPS can significantly improve the baselines in terms of accuracy and robustness. Note that we have not fine-tuned our hyperparameters for GPS in the tables yet. In summary, our GPS can be applied to a wide range of applications for performance improvement.

We thank the reviewer for the valuable comments. Below are our responses to your concerns:

1. Generalized Extreme Value (GEV) Distribution vs. Gumbel Distribution: As mentioned, GEV is a generalization of Gumbel distributions by incorporating additional hyperparameters. In our research, we have tested various GEV distributions with different hyperparameters (though these experiments were not included in the submission). We found that they did not significantly improve performance compared to the Gumbel distribution but only increased the fine-tuning time. Therefore, to maintain simplicity, we chose the Gumbel distribution as our prior distribution for fitting data. We will add such discussion in our final paper.

2. Modelling ordinary order statistics using Gumbel: Typically, the Gumbel distribution is used to model minima or maxima, such as first-order nearest neighbors in our case. However, we would like to emphasize the following points:

• (a) Theoretically, it has been shown that under certain conditions, Extreme Value Theory (EVT) can be applied to (generalized) order statistics (e.g., Nasri-Roudsari, Dirk. "Extreme value theory of generalized order statistics." Journal of Statistical Planning and Inference 55.3 (1996): 281-297).

• (b) Empirically, we have demonstrated that it is possible to fit KNN data with Gumbel distributions, as shown in Fig. 2(c).

• (c) Our approach is similar to Gaussian Mixture Models (GMM), where GMM uses Gaussian as a prior, while our GPS uses Gumbel as a prior. The actual distribution of the data (whether Gaussian or Gumbel) is less important as long as the problem can be effectively addressed based on the model.

3. Related References: Thank you for providing these references; we will include them in our paper. However, the main point we want to emphasize is that we are the first to introduce the Gumbel similarity as a set-to-set loss function in deep learning, rather than using it for matching. To the best of our knowledge, no existing work has done this, and we would appreciate any feedback you may have.

4. Experiments: A closer reading of our submission, specifically lines 152-155, reveals that SnowflakeNet, PoinTr, and SeedFormer are all transformer-based networks for point cloud completion, and our GPS consistently enhances these baseline approaches.

Moreover, we conducted more experiments for both point cloud matching/registration and image matching using $GPS(\alpha, \beta)$ with only 1-NN and 1 Gumbel distribution. Details are as follows:

• (a) Point Cloud Matching with PREDATOR (Huang et al. CVPR, 2021): We modified the public code from https://github.com/prs-eth/OverlapPredator by simply replacing the distances in the loss with the negative GPS scores. The results are listed below:

| Samples | 5000 | 2500 | 1000 | 500 | 250 | Ave.$\pm$Std |

| PREDATOR | 89.0 | 89.9 | 90.6 | 88.5 | 86.6 | 88.92$\pm$1.53 |

| P + GPS(1, 1) | 89.3 | 89.6 | 90.0 | 88.2 | 88.7 | 89.16$\pm$0.72 |

| P + GPS(1, 2) | 88.6 | 90.2 | 90.3 | 89.2 | 88.4 | 89.34$\pm$0.88 |

| P + GPS(2, 2) | 89.4 | 89.4 | 90.8 | 89.3 | 89.3 | 89.64$\pm$0.65 |

• (b) Image Matching with LightGlue (Lindenberger et al. CVPR, 2023): Replacing distances in the loss for SuperGlue (Sarlin et al. CVPR, 2020) with our negative GPS scores is not straightforward, because there are some numerical stability issues in the log-space. Instead, we utilized LightGlue (https://github.com/cvg/LightGlue), which is inherited from SuperGlue, for image matching. Same as we did for PREDATOR, we replaced the feature distances with our negative GPS scores for computing the loss in LightGlue. The results are listed below:

| AUC | 5 degree | 10 degree | 20 degree | Ave.$\pm$Std |

| LightGlue | 41.5 | 57.8 | 70.7 | 56.67$\pm$14.63 |

| L + GPS(1, 1) | 42.5 | 48.3 | 71.2 | 57.33$\pm$14.37 |

As we can see, in both applications, our GPS can significantly improve the baselines in terms of accuracy and robustness. Note that we have not fine-tuned our hyperparameters for GPS in the tables yet. In summary, our GPS can be applied to a wide range of applications for performance improvement.

5. Usage of $\delta$ from a Theoretical View: Theoretically, the constant $\delta$ influences the mode of a Gumbel distribution, as shown in Fig. 4. As mentioned in lines 258-260, "In many applications, such as point cloud completion, the objective is to minimize the distances between predictions and the ground truth." In these applications, this constant distance shift is crucial for learning.

Consider the PDF of a Generalized Extreme Value distribution, $GEV(\mu, \sigma, \xi)$, as follows:

$$f(x) = \frac{1}{\sigma} t(y)^{\xi+1} \exp(-t(y)),$$

where $y = \frac{x - \mu}{\sigma}$, $t(y) = \exp(-y)$ if $\xi=0$, otherwise, $t(y) = (1+\xi y )^{-\frac{1}{\xi}}$. Here, $x\in\mathbb{R}$ denotes a random variable, and $\mu\in\mathbb{R}, \sigma>0, \xi\in\mathbb{R}$ are three parameters for the GEV. Note that the shape parameter $\xi$ controls the distribution family, i.e., $\xi=0$ leads to a Gumbel distribution, $\xi>0$ leads to a Fréchet distribution, and $\xi<0$ leads to a Weibull distribution.

We reran the experiments in Table 4 for few-shot image classification on CIFAR-FS dataset, with $\mu=0.8, \sigma=0.5$ (equivalently $\alpha=5, \beta=2$ in our submission). The table below summarizes the comparison results between distributions:

| Weibull | Gumbel | Fréchet |

| $\xi=-1$ | $\xi=-0.1$ | $\xi=-0.01$ | $\xi=0$ | $\xi=0.01$ | $\xi=0.1$ | $\xi=1$ |

| 71.29 | 73.18 | 73.47 | 74.22 | 73.65 | 73.31 |71.78 |

The trend of such numbers is clear that the accuracy seems higher towards $\xi=0$, i.e. Gumbel. We hope that such results can convince the reviewer that Gumbel distributions may be a better choice to fit the nearest neighbors than the other two distributions for learning, not only higher accuracy but also fewer parameters.

We thank the reviewer for their thoughtful feedback and for encouraging deeper consideration of alternative distributions. First of all, we need to highlight that Figure 2 indeed plots the distributions, which the reviewer may have missed looking at. Below, we clarify why the Gumbel distribution remains the most suitable choice for GPS and address the reviewer’s suggestions:

1. Empirical Evidence from Figure 2: Figure 2 shows the histograms of 1-NN, 2-NN, and 3-NN distances exhibit:

• Skewed, heavy-tailed behavior, which aligns naturally with the Gumbel distribution.

• Zero derivatives near zero, reflecting a smooth plateau that is not captured by distributions like Gamma or Erlang, which have positive slopes near zero.

2. Why Not Gaussian, Exponential, or Gamma:

• Gaussian: Assumes symmetry and light tails, which contradicts the observed data’s skewness and heavy tails.

• Exponential: Lacks flexibility to model broader variability or smooth behavior near zero.

• Gamma/Erlang: While flexible for skewed data, these distributions are not designed for modeling extrema and cannot reproduce the zero-derivative property without significant parameter tuning.

3. Why Gumbel:

• The Gumbel distribution is explicitly designed for modeling extrema and aligns with the observed characteristics of the data, including skewness, heavy tails, and smoothness near zero.

• Its differentiable parameterization allows backpropagation through extrema, enabling efficient gradient-based optimization in neural network training.

4. Additional Results: We have run some distributions on point cloud completion using the setting in Table 8, and list the results below:

| Distribution | L2-CD$\times 10^3$ |

| Weibull | 4.09 |

| Normal | 4.17 |

| Logistic | 4.18 |

| Log-Logistic | 4.10 |

| Chi-Squared | 4.68 |

| Gamma | 4.22 |

| GPS (Gumbel) | 3.94 |

We further run some distributions on few-shot image classification using the setting in Table 1, and list the results below:

| Distribution | Accuracy (%) |

| Weibull | 72.59 |

| Normal | 72.85 |

| Logistic | 72.59 |

| Log-Logistic | 72.64 |

| Chi-Squared | 72.89 |

| Gamma | 72.65 |

| GPS (Gumbel) | 74.22 |

We believe that this additional analysis, combined with the evidence in Figure 2, should be able to further clarify why Gumbel remains the most effective choice for GPS. We appreciate the reviewer’s feedback and look forward to further input.

Dear Reviewer,

To address your concerns, we have included additional results using various distributions for both few-shot image classification and point cloud completion. Our GPS consistently performs best across these tasks. These results, along with our discussion on different distribution choices, have clearly demonstrated our novelty.

Is there anything else you would like us to address, as today is the rebuttal deadline? Thank you.

Thank you for your valuable comments! Below are our responses to your questions:

1. Hyperparameter Tuning: In fact, we found that this issue is moderate in practice for different applications, because (1) we can always choose the nearest neighbors for GPS and (2) the ranges for $\alpha, \beta$ are often very limited.

2. Finding correspondences: This process is automatically handled by the nearest neighbor search when computing GPS, requiring no additional steps. As training with GPS progresses, we anticipate that the correspondences between the sets will improve over time, resulting in enhanced performance.

3. Point cloud registration: We conducted more experiments for both point cloud matching/registration and image matching using $GPS(\alpha, \beta)$ with only 1-NN and 1 Gumbel distribution. Details are as follows:

• (a) Point Cloud Matching with PREDATOR (Huang et al. CVPR, 2021): We modified the public code from https://github.com/prs-eth/OverlapPredator by simply replacing the distances in the loss with the negative GPS scores. The results are listed below:

| Samples | 5000 | 2500 | 1000 | 500 | 250 | Ave.$\pm$Std |

| PREDATOR | 89.0 | 89.9 | 90.6 | 88.5 | 86.6 | 88.92$\pm$1.53 |

| P + GPS(1, 1) | 89.3 | 89.6 | 90.0 | 88.2 | 88.7 | 89.16$\pm$0.72 |

| P + GPS(1, 2) | 88.6 | 90.2 | 90.3 | 89.2 | 88.4 | 89.34$\pm$0.88 |

| P + GPS(2, 2) | 89.4 | 89.4 | 90.8 | 89.3 | 89.3 | 89.64$\pm$0.65 |

• (b) Image Matching with LightGlue (Lindenberger et al. CVPR, 2023): Replacing distances in the loss for SuperGlue (Sarlin et al. CVPR, 2020) with our negative GPS scores is not straightforward, because there are some numerical stability issues in the log-space. Instead, we utilized LightGlue (https://github.com/cvg/LightGlue), which is inherited from SuperGlue, for image matching. Same as we did for PREDATOR, we replaced the feature distances with our negative GPS scores for computing the loss in LightGlue. The results are listed below:

| AUC | 5 degree | 10 degree | 20 degree | Ave.$\pm$Std |

| LightGlue | 41.5 | 57.8 | 70.7 | 56.67$\pm$14.63 |

| L + GPS(1, 1) | 42.5 | 48.3 | 71.2 | 57.33$\pm$14.37 |

As we can see, in both applications, our GPS can significantly improve the baselines in terms of accuracy and robustness. Note that we have not fine-tuned our hyperparameters for GPS in the tables yet. In summary, our GPS can be applied to a wide range of applications for performance improvement.