TopoFR: A Closer Look at Topology Alignment on Face Recognition

We thank the Reviewer GA4X for the careful reading of the manuscript and the related comments, which are helpful to improve our paper. Our detailed point-by-point responses are provided below.

W1: It is not clear if this method can be applied to recent proposed loss AdaFace.
A1: According to your suggestion, we conduct additional experiments on AdaFace model to further validate the universality of our method, as shown in the following Table R1.
In comparison to the baseline model ArcFace, AdaFace is a highly advanced FR model. Consequently, integrating our method with AdaFace can notably enhance the model's verification accuracy across multiple benchmarks. These improved results validate the universality of our method.

Table R1: Verification performance (%) on IJB-C, IJB-B, CPLFW and CALFW.

W2: Includes comparison of the model's GFLOPs.
A2: Based on your valuable advice, we provide a comparison of model's GFLOPs, as shown in the following Table R2. As can be seen, our TopoFR has the same GFLOPs as the vanilla ArcFace model (Baseline) and some existing popular FR models (e.g., MagFace, AdaFace and CurricularFace), since we adopt the same network architecture.

Table R2: Comparison of Model's GFLOPs.

Q1: In Eq.(7), how to determine the initial values of parameters $\phi$, $\Sigma$, and $\Omega$ in the GUM statistical distribution ?
A3: We refer to prior work [S4] to set the initial values for the EM algorithm used in our GUM parameter estimation process.

Ref.[S4] Using gaussian-uniform mixture models for robust time-interval measurement. TIM 2015.

Q2: In Eq.(2), does the ISA function loss $\mathcal{L}_{sa}$ guide the optimization of the entire network or only specific components of the network ?
A4: In fact, the ISA loss $\mathcal{L}_{sa}$ is only responsible for optimizing the parameters of the feature extractor $\mathcal{F}$.

As stated in Section 4.1 (lines 140-146), since computing the pairwise distance matrices $\mathcal{M}^{\mathcal{X}}$ and $\mathcal{M}^{\mathcal{\widetilde{Z}}}$ for Vietoris-Rips (VR) complexes \mathcal{V}\_\{\rho}(\mathcal{X}) and \mathcal{V}\_\{\rho}(\mathcal{\widetilde{Z}}) is a differentiable process, the ISA loss $\mathcal{L}_{sa}$ will perform gradient back-propagation through the two distance matrices to optimize the parameters of the feature extractor $\mathcal{F}$.

Q3: What distance criterion is used to construct the topological structure space of the data in Vietoris-Rips complex during model forward propagation ?
A5: As stated in Section 3 (lines 95-104), we utilize the Euclidean distance as the distance criterion to construct the Vietoris-Rips (VR) complex $\mathcal{V}_{\rho}$ and analyze the topological structure of the underlying space.

Q4: In ISA loss, what dimension of topological holes does the persistent homology (PH) capture and analyze ?
A6: As described in Section A.1 of the Appendix (lines 547-550), we preserve and analyze 0-dimension topological holes $H_{0}$ (e.g., connected components) in our ISA loss $\mathcal{L}_{sa}$. Because some preliminary experiments have shown that using the 1-dimension or higher-dimension topological holes only increases model’s training time without bringing clear performance gains.

Q5: I am very curious about the reasons why the structure differences between the input pixel and latent spaces decrease as the network becomes deeper.
A7: This is a very interesting question. When we obtained this finding in the experiment, we also contemplated its underlying reasons.
In our opinion, the deep neural network is actually a complex function mapping process, and each network layer can be regarded as a sub-function in this composite function. If the neural network is deeper, the mapping of each network layer will become smoother, and the data will lose less information during the mapping process. On the contrary, if the neural network is shallower, the mapping of each network layer will become sharper, and some intrinsic information of the data, such as structure information, is easily destroyed during the mapping process. Similar conclusions can also be found in the flow-based models [S5].

Ref. [S5] Variational inference with normalizing flows. ICML 2015.

Dear Reviewer GA4X:

Thank you for the positive feedback. We appreciate your efforts in reviewing our work. We will reflect your suggestions in the revised version to enable it to be a high-quality paper.

Best Regards, Authors of 505.

We appreciate Reviewer 68Et for the thorough review of the paper and valuable comments that will aid in enhancing our paper.

W1: Formula 7 may require a more specific explanation.
A1: Based on your valuable suggestion, we provide an explanation of how to estimate the parameter set $\varphi = \\left \\{ \pi, \Sigma, \Omega \\right \\}$ of the Gaussian-uniform mixture (GUM) model using EM algorithm.

First, we transform the original GUM model (Eq.(3)) into a more easily solvable statistical distribution (Eq.(6)) using the Bernoulli distribution-based sampling method. In this way, the maximum likelihood model of Eq.(6) is formulated as: \max\limits\_\{\pi,\Sigma,\Omega} \displaystyle\prod_{i=1}^{n}p\left(\widehat{E}(\widetilde{g}\_\{i})|\widetilde{x}\_\{i}\right ). Then, we can use the EM algorithm to solve the maximum likelihood problem in order to estimate the parameter set $\varphi$ of GUM. The specific iterative updating formulas of EM algorithm are shown in Eq.(7).

Specially, at each iteration, EM alternates between evaluating the expected log-likelihood (E-step) and updating the parameter set $\varphi$ (M-step). In Eq.(7), the E-step aims to evaluate the posterior probability $h_{\varphi}^{(t+1)}$ of an sample \widetilde{x}\_\{i} to be hard sample using the iterative formula h\_\{\varphi}^{(t+1)}(\widetilde{x}\_\{i}), where $(t+1)$ denotes the EM iteration index. The M-step updates the parameter set $\varphi$ using the iterative formulas $\pi^{(t+1)}$, $\Sigma^{(t+1)}$ and $\Omega^{(t+1)}$, where $\eta_{1}$ and $\eta_{1}$ are the first-order and second-order centered data moments, respectively.

W2: The meaning of H0 in Figure 1, along with the distribution style and other differences, is best explained.
A2: As described in Sec.3 (lines 105-109), homology is an algebraic structure that analyzes the topological features of a simplicial complex in different dimension $j$, including connected components ($H_{0}$), cycles ($H_{1}$), voids ($H_{2}$), and higher-dimensional topological features ($H_{j},j\geq 3$). By tracking the changes in topological features $H_{j}$ across different dimensions $j$ as the scale parameter $\rho$ increases, we can obtain the multi-scale topological information of the underlying space.

Thus, $H_{0}$ is the $0$-th dimension homology in the persistence diagram, which captures the $0$-th dimension topological feature of the underlying space. Notably, low-dimension topological features (e.g., $H_{0}$) can roughly reflect the topological structure of the space, while high-dimension topological features (e.g., $H_{3}$, $H_{4}$) aim to capture the intricate details of the space's topological structure.

Moreover, in topology theory, an increase in the number of high-dimensional topological features within a space corresponds to a more complex topological structure. As illustrated in Figs 1(a)-1(d), as the amount of face data increases, the persistence diagram contains more and more high-dimensional homology (e.g., $H_{3}$ and $H_{4}$), indicating that the input space contains an increasing number of high-dimensional topological features. Thus, this phenomenon shows that the topological structure of the input space is becoming more and more complex.

W3: The hard sample mining strategy SDE seems to have little correlation with Perturbation-guided Topological Structure Alignment (PTSA).
A3: As stated in Sec.1, the large-scale face dataset contains rich topological structure information. However, we find that these structure information is not effectively preserved in the latent space of existing FR models, as verified in Fig 2 (lines 31-43). Existing studies on FR have overlooked this issue, which limits the generalization of FR models. To solve this problem, we propose the PTSA strategy. Specially, our PTSA strategy directly extracts the topological structure information in the face dataset from the input pixel space and then encodes these powerful structure information into the latent space by aligning the topological structures of the input and latent spaces.

However, as stated in Sec 4.2 (lines 170-177), we experimentally find that some low-quality samples (e.g., hard samples) can easily be encoded to abnormal positions in the latent space during training, which disrupts the latent space’s topological structure and further hinder the alignment of topological structures. Thus, we propose the SDE strategy to identify hard samples with significant structure damage and effectively mitigate their adverse impact on the latent space’s topological structure by guiding them back to their appropriate positions during optimization.

Hence, our PTSA strategy and SDE strategy are complementary to each other. Additionally, the ablation study in Table 3 also validate the complementarity of two strategies.

Q1: Is this topological alignment strategy only for face recognition (FR) tasks?
A4: In addition to FR systems, the proposed PTSA strategy can also be applied to Image Retrieval and Large Language Model (LLM), since these tasks all require large-scale datasets for training.
Existing works on Image Retrieval and LLM seem to neglect the utilization of topological structure information hidden in large-scale datasets. Thus, we believe that applying the topological structure alignment technique to these tasks can effectively improve the representation power of features and the generalization ability of models.

Q2: Is the process of alignment computationally intensive?
A5: In fact, we have provided an analysis of the model training time in our manuscript (lines 637-650). Please refer to Table 13 in the Appendix. For example, compared to the R50 ArcFace (Baseline), our R50 ArcFace + PTSA requires about 2.24 seconds more training time per 100 steps. These results indicate that introducing PTSA strategy leads to significant performance gains with only a small increase in training time, which is acceptable.

We sincerely appreciate Reviewer SWuk for the careful reading and the insightful comments, which are helpful in improving our paper. Our detailed point-by-point responses are provided below.

W1: The results of the method on IJB-B and IJB-C are minor.
A1: Due to the characters limit, we will respond to you in the global rebuttal area.

W2: The technical approach seems to be an incremental one. The PTSA strategy is similar to the Gromov-Wasserstein distance.
A2: Due to the characters limit, we will respond to you in the global rebuttal area.

W3: I am not sure whether the PTSA strategy significantly improves the structure information or not. Why the authors claim the proposed approach improve the topological structure of the feature space? It would be better to visualize the feature distributions.
A3: As shown in Figures 2(a) and 2(b), we experimentally find that there are significant topological structure discrepancy between the input space and the latent space of existing FR models. This observation indicates that the topological structure information of the large-scale dataset is not effectively preserved in the latent space and may even be destroyed. Therefore, we propose the PTSA strategy to align the topological structure of two spaces. The visualization results in Figs 2(d), 5 and 7 have validated that with the help of our PTSA strategy, the topological structure discrepancy between two spaces have significantly decreased. In fact, the results in Figs 2(d), 5 and 7 can quantify topological structure discrepancy more objectively and accurately than t-SNE. Moreover, the ablation study results in Tab 8 of Appendix have also validated that the addition of PTSA can implicitly enhance the intra-class compactness and inter-class separability of facial features.

t-SNE: Based on your insightful advice, we sample 10 identities face images from MS1MV2 dataset and utilize the t-SNE to visualize the facial features learned by ArcFace and ArcFace + PTSA. The results are shown in the Figure 1 of the global response PDF file. We can observe that using the proposed PTSA strategy to reduce the topological structure discrepancy between two spaces can greatly enhance the discriminative power of the learned facial features (e.g., with better inter-class separability and intra-class compactness), thereby boosting the FR model's generalization performance. The t-SNE results are consistent with our aforementioned experimental results.

W4: Other methods are also considered as approaches to improve facial features distributions in latent space via well design loss functions.
A4: Existing FR methods, such as ArcFace, CosFace, SphereFace and CurricularFace, tended to design some margin-based softmax loss functions to explicitly encourage facial features with better inter-class separability and intra-class compactness in feature space. However, the experimental results in Figs 2(a) and 2(b) have shown that the latent space's topological structure of existing models may be destroyed, which limits FR model's generalization. Thus, unlike these existing FR works that directly force features to become more discriminative in the feature space, our PTSA strategy aims to leverage the intrinsic topological structure information in training dataset (i.e.,from the input pixel space) to guide the construction of the latent space's topological structure and the learning of facial features. Detailed experiments have indicated that our method can effectively reduce topological structure discrepancy and boost model's generalization.

In general, leveraging the topological structure information in dataset to enhance the model's generalization is a different research direction from designing the margin-based softmax loss functions to improve facial features distribution in the deep feature space.

Q1: Are the persistence diagrams (PDs) and persistence pairings (PPs) computation in ISA loss differentiable for backpropagation? Detail this computational process.
A5: Yes, the computation of PDs and PPs in ISA loss is differentiable for back-propagation.

1) Detailed computation process: Persistent homology (PH) is a method used to analyze the topological structure information of complex point clouds (e.g., mini-batch data). In the original input space, we first flatten the mini-batch face images into vector features to model point clouds $\mathcal{X}$, and then calculate their pairwise distance matrix to construct the Vietoris-Rips (VR) complex \mathcal{V}\_\{\rho}(\mathcal{X}). Similarly, we can construct another VR complex \mathcal{V}\_\{\rho}(\widetilde{\mathcal{Z}}) for the perturbed latent features $\widetilde{\mathcal{Z}}$. After that, we use the Ripser package [S3] to implement the PH to extract topological features from two VR complexes, and obtain their corresponding PDs and PPs. The discrepancy between the input space's PD and latent space's PD represents the topological structure discrepancy of two spaces. However, directly calculating the distance between two PDs is computationally expensive. We thus turn to utilize PPs to estimate the discrepancy between two PDs. PPs contain indices of simplices that are related to the birth and death of the topological feature. In this case, we can calculate the discrepancy between two PDs by utilizing PPs to retrieve the differences between two pairwise distance matrices.

Ref.[S3] Ripser: Efficient Computation of Vietoris–Rips Persistence Barcodes. 2021.

2) Optimization of model params w.r.t $\mathcal{L}_{sa}$: Please refer to the 6th answer A4 to Reviewer GA4X.

Q2: Discuss the broader impact.
A6: Due to the characters limit, we will respond to you in the global rebuttal area.