ECLayr: Fast and Robust Topological Layer based on Differentiable Euler Characteristic Curve

We appreciate your valuable comments and questions. We address each of them below.

1. [Concern on gradient approximation] Thank you for your thorough feedback. First, we would like to clarify that our approach is not equivalent to approximating the indicator function with a linear function $\gamma (f_\sigma - t)$. Specifically, the gradient in our method produces a constant value $\gamma$ at $t=f_\sigma$ (or at $t_{i+1}$ after shifting, given $f_\sigma \in (t_i, t_{i+1}]$) and zero elsewhere, whereas the gradient of the linear function is $\gamma$ for all values of $t$. Nevertheless, you are absolutely correct in stating that the magnitude of the gradient at $t = f_\sigma$ is always fixed to a constant $\gamma$ in our approach. While the simplicity of our method may initially appear to limit its novelty, it sufficiently addresses key issues associated with sigmoid approximation. Primarily, it resolves the gradient inconsistency and vanishing problems, as detailed in Section 4 of our paper. Additionally, our backpropagation technique facilitates the use of Algorithm 1 during the forward pass, achieving enhanced computational efficiency with a time complexity of $O(N+v)$ compared to the $O(vN)$ required by the sigmoid approximation; the $O(vN)$ time complexity of ECC computation using sigmoid approximation arises from the need to apply the sigmoid function to all $t_i \in \text{tseq}$ during each iteration across all $\sigma \in K$.

2. [Concern on shifting] Thank you for this insightful question. First, we would like to clarify that although the gradient at $t = f_\sigma$ is fixed to a constant $\gamma$, the information of filtration value is not lost. This information is incorporated into the backpropagation location $t_{i+1}$ through the dependence $t_i < f_\sigma \leq t_{i+1}$. Similarly, for the sigmoid approximation with a very large $\lambda$, the gradient is also fixed to a constant $\lambda/4$ at $t = f_\sigma$ and nearly zero elsewhere. In both cases, the filtration value is not encoded directly in the gradient itself but rather in the location $t = f_\sigma$ to which the gradient is backpropagated. In fact, sigmoid approximation is more prone to information loss due to potential gradient inconsistency or vanishing, where parts or even the entirety of the gradient information can be lost. Second, we completely agree with your observation regarding the distortion of topology information. However, this distortion primarily occurs during the forward pass when ECC is approximated as a vector. Specifically, the jump of the indicator $I(f_\sigma \leq t)$ is reflected at $t_{i+1}$ rather than at $t=f_\sigma$, given $t_i \leq f_\sigma \leq t_{i+1}$. This type of distortion is inherent to all vector approximation methods, including PLLay and DECT. Since we backpropagate the gradient to $t_{i+1}$, the exact location where the jump was reflected in the forward pass, we believe no additional distortion is introduced during backpropagation. Moreover, provided that the discretization is not excessively sparse, this distortion is unlikely to significantly affect the global topology of data.

3. [Stability result]: While smaller $\Delta t$ allows for a more accurate approximation of ECC, we would like to emphasize that using a highly dense discretization is not necessarily beneficial. Given the discretized interval, the vectorized ECCs are not able to capture cycles that are born and perished between $t_i$ and $t_{i+1}$. However, cycles with very short life spans are often insignificant (noisy) generators. Therefore, excessively dense discretization may not be beneficial, as including numerous small, noisy generators in the learning process can lead to increased variance (please refer to the illustration provided in Figure 6 of the revised manuscript). By avoiding excessively fine discretization, we can prevent overly loose stability bounds in Proposition 5.1. However, we acknowledge that the stability results of ECC are less strict compared to PH-based descriptors, such as persistence images, as noted at the end of Section 5.

4. [Comparison with naive representations]: This is a great question. A naive representation, such as a vector of top-50 largest filtration values, fails to preserve meaningful topological and geometrical information. For instance, the loop structure in digit 0 of the MNIST dataset is lost in such a representation. In contrast, our proposed representation captures the Euler characteristic of the underlying structure within the data, thereby effectively preserving topological and geometrical information.

5. [Ans to Q1]: Deep learning models are renowned for their ability to autonomously extract meaningful features from data during training. In a similar vein, we anticipate that $g_\theta$ will effectively retain relevant information while filtering out noise that may not contribute to the task.

Thank you for your valuable comments and suggestions. We address each of them below.

1. [Novelty of the work]: Thank you for raising this point. While ECC itself is not a new concept, we would like to highlight that the development of a differentiable topological layer based on ECC is novel. The idea of differentiating ECC with respect to the layer input has only been previously explored in [1], which was limited to the height filtration and lacked detailed theoretical analysis. In contrast, our work introduces a general framework that supports any differentiable filtration and is accompanied by theoretical stability results. Furthermore, we propose a novel backpropagation method that addresses the gradient inconsistency issue caused by the sigmoid approximation used in [1]. By removing the need for sigmoid approximation, our approach also improves the forward pass’s time complexity to $O(N+v)$, compared to the $O(vN)$ complexity associated with the sigmoid-based method; the $O(vN)$ time complexity of ECC computation using sigmoid approximation arises from the need to apply the sigmoid function to all $t_i \in \text{tseq}$ during each iteration over all $\sigma \in K$. Consequently, our primary contribution is not simply leveraging ECC for computational efficiency, but introducing a comprehensive framework for differentiable ECC. This framework includes a novel backpropagation method and rigorous theoretical analysis of both stability and backpropagation.

2. [Stability comparison with PH-based methods]: At the end of Section 5, we briefly mentioned that our stability results are less strict compared to those of PH. Below, we provide a direct comparison for clarity. By combining Proposition 5.1. and 5.2., the output of our layer is bounded by

whereas the output of [2](Theorem 4.1.) is bounded by

The results indicate that PH-based methods are generally bounded by the bottleneck distance, whereas our layer adopts a more relaxed bound based on the Wasserstein distance. We hope this offers a clear comparison of the stability results between ECC and PH-based methods.

3. [Ans to Q1 and experiment on larger dataset]: Thank you for your suggestion. In response, we conducted an additional experiment on ORBIT5K, a widely used point cloud benchmark dataset in TDA. The data contains 5000 samples, offering a relatively larger sample size compared to our previous experiments. In this experiment, we introduced increased noise levels of up to $30$%. Due to limited computational resources, the experiment was conducted only on our ECC based models. The results were on par with the numbers reported in [2] up to a 10% noise level, but showed a significant drop in performance beyond this threshold.

4. [Ans to Q2]: ECLayr introduces four TDA hyperparameters: filtration, interval $[T_{min}, T_{max}]$, discretization $v$, and gradient control $\beta$. The initialization procedure of ECLayr proceeds as follows. First, a filtration is constructed by selecting an appropriate simplicial complex $K$ and function $f: K \rightarrow \mathbb{R}$. Next, an interval $[T_{min}, T_{max}]$ is chosen as the domain on which ECC will be computed. This interval is then discretized into $v$ evenly-spaced grid points. The vectorized ECC can subsequently be computed using Algorithm 1. During backpropagation, $\beta$ controls the magnitude of the gradient, with smaller values of $\beta$ yielding stronger gradients. For your reference, the hyperparameter settings used in our experiments are listed below and can also be found in Experiment Details section of the Appendix.

MNIST - data scarcity

• filtration: superlevel set filtrations on T-constructed cubical complex
• $[T_{min}, T_{max}]$: $[0,1]$
• $v$: 32
• $\beta$: 0.01

MNIST - data contamination

• filtration: DTM filtrations with $m_0=0.05$ and $m_0=0.2$, respectively
• $[T_{min}, T_{max}]$: $[0.02, 0.28]$ and $[0.06, 0.29]$ for $m_0 = 0.05$ and $m_0 = 0.2$ respectively
• $v$: 32
• $\beta$: 0.01

Br35H

• filtration: superlevel set filtrations on V-constructed cubical complex
• $[T_{min}, T_{max}]$: $[0.4, 1]$ for first ECLayr placed in parallel to ResNet and $[0.3, 1]$ for second ECLayr placed between max pool and first residual layer of ResNet.
• $v$: 64
• $\beta$: 0.01

Roell, Ernst, and Bastian Rieck. "Differentiable Euler characteristic transforms for shape classification." [1]

Kwangho Kim, Jisu Kim, Manzil Zaheer, Joon Kim, Frédéric Chazal, and Larry Wasserman. "Pllay:Efficient topological layer based on persistent landscapes". [2]

We appreciate your valuable input and insights. We address each of your comments below.

1. [Additional experiments] Thank you for your suggestion. We completely agree that additional experiments on other data modalities would strengthen our work. In our future revised paper, we shall conduct additional experiments on the ORBIT5K point cloud data, a dataset that is often used as a benchmark in TDA.

2. [Ablation study]: We appreciate your suggestion on this matter. In response, we conducted an ablation study and included the results in Appendix I.5 of our revised paper.

3. [Ans to Q1]: We believe incorporating TDA features is most effective when the baseline model struggles to learn sufficient information from data. By intentionally restricting the size of data to 100 samples, we created a scenario where the baseline CNN encounters difficulties in learning adequate information. In such cases, the additional topological information provided by ECC notably improved model performance and generalization capabilities. Conversely, when the baseline model can effectively extract sufficient information from the data, the impact of incorporating TDA features may be minimal. Given the simplistic and macroscopic geometrical features of the MNIST dataset, the baseline CNN quickly captured enough information as the sample size increased, leading to a reduction in performance gain. When the dataset comprises more complex topological and geometric features that the baseline model struggles to capture, we conjecture that incorporating TDA features will lead to improvement in performance and generalizability, even as the training set size increases.

4. [Ans to Q2]: This is a good point. The MNIST dataset contains images of hand-written digits, each having at most 2 loop structures (e.g., digit 8 has 2 loops). To effectively capture these topological features, we selected the $top2$ function as the permutation invariant function for both models. Notably, the $top2$ function was also used in the MNIST experiment of the original PLLay paper[1] for the same purpose.

Kwangho Kim, Jisu Kim, Manzil Zaheer, Joon Kim, Frédéric Chazal, and Larry Wasserman. "Pllay:Efficient topological layer based on persistent landscapes". [1]

Thank you for your time and valuable feedback. We would like to address each of your major concerns in the following.

1. [Investigation on features captured by ECLayr]: ECLayr captures the Euler characteristic of the underlying structure within the data. To facilitate understanding, we have included a visual illustration in Figure 6 of the updated manuscript. We hope this addition enhances the clarity of our work.

2. [Presentation of hyperparameter selection process]: Thank you for your valuable feedback. We fully agree that additional clarification on hyperparameter selection would strengthen our paper. To address this, we have included a new section in the Appendix of the amended manuscript, providing guidelines and insight on this topic. Below, we briefly summarize the key points. For a comprehensive explanation, including illustrations and examples, we kindly refer you to Appendix H of the revised paper.

• Choice of filtration: The choice of filtration is fairly straightforward, as certain filtrations are commonly favored for specific data modalities and training contexts. Ultimately, the filtration that best captures the topological features of the given data should be selected.

• Choice of $[T_{min}, T_{max}]$. A naive and convenient approach is to assign $T_{min}$ and $T_{max}$ as the minimum and maximum of possible filtration values, respectively. Alternatively, one could select $[T_{min}, T_{max}]$ as a tighter interval within the range of possible filtration values, focusing on regions of the filtration that contain meaningful topological and geometrical information. Such an interval can be identified via hyperparameter search, or chosen intuitively by examining the ECC computed for some data.

• Choice of $v$: The spacing between grid points is important as our vectorized ECC does not account for cycles that are born and dead between $t_i$ and $t_{i+1}$, where $t_i, t_{i+1} \in tseq$. Provided that the discretization is not overly sparse, the uncaptured cycles are often small (noise) generators with life span shorter than $t_{i+1} - t_i$. This implies that with appropriate discretization, noise can be partially filtered by design. Therefore, using a highly dense discretization is not necessarily beneficial, as it captures even the small (noise) generators. Conversely, using a excessively sparse discretization may jeopardize the capturing of essential global features. The optimal choice of $v$ is not always evident; we recommend hyperparameter search using cross validation to determine the adequate $v$ that balances the two circumstances.

• Choice of $\beta$: The optimal choice of $\beta$ is somewhat ambiguous. Unfortunately, we do not have a clear rationale for choosing $\beta$; it is contingent upon numerous factors, including model architecture and the specific task at hand. Therefore, we recommend conducting hyperparameter search via cross validation to select an appropriate $\beta$.

3. [Ans to Q1]: We appreciate your suggestion on this matter. In response, we conducted an ablation study and included the results in Appendix I.5 of our revised paper.

4. [Ans to Q2]: This is because, for certain datasets, capturing the major macroscopic geometrical features is often sufficient for the classification task. In such cases, relying on more complex, high-resolution features can lead to overfitting. We further conjecture that the relative simplicity of ECC facilitates the optimization process compared to the more intricate features derived from PH, particularly in scenarios with limited data, thereby yielding improved performance. We note that, however, if the topological features at a more granular level are critical for the learning task, PH-based features are likely to yield superior performance. We will clarify this in the revised text.

We hope the above response addresses your concerns. However, if there are any remaining issues, please feel free to let us know through your comments.

13. [Ans to Q13]: Our motivation was purely data-driven, as we conducted a hyperparameter search to identify the training hyperparameters that yield the best performance. Initially, we searched for the optimal training hyperparameters for the vanilla CNN/ResNet models without topological layers. Once these were determined, we attached the topological layers and performed a separate hyperparameter search for the topological layers, keeping the CNN/ResNet hyperparameters fixed to the previously identified optimal values.

We hope the above response addresses your concerns. However, if there are any remaining issues, please feel free to let us know through your comments.

[1] Kwangho Kim, Jisu Kim, Manzil Zaheer, Joon Kim, Frédéric Chazal, and Larry Wasserman. "Pllay:Efficient topological layer based on persistent landscapes".

[2] Roell, Ernst, and Bastian Rieck. "Differentiable Euler characteristic transforms for shape classification".

[3] Mathieu Carriére, Frédéric Chazal, Yuichi Ike, Théo Lacombe, Martin Royer, and Yuhei Umeda. "Perslay: A neural network layer for persistence diagrams and new graph topological signatures".

8. [Ans to Q8]: We absolutely agree that a visualization of the model architectures would enhance clarity and understanding of our work. Accordingly, we have included illustrations in Figure 8 of the revised manuscript. Regarding performance of ECC and PH, we speculate that the observed gap mainly arises from the inherent nature of the two descriptors, rather than differences in their gradients. For certain datasets, capturing the major macroscopic geometrical features is often sufficient for the classification task. In such cases, relying on more complex, high-resolution features can lead to overfitting. We further conjecture that the relative simplicity of ECC facilitates the optimization process compared to the more intricate features derived from PH, particularly in scenarios with limited data, thereby yielding improved performance. We note that, however, if the topological features at a more granular level are critical for the learning task, PH-based features are likely to yield superior performance. We will clarify this in the revised text.

9. [Ans to Q9]: Thanks for this question too. The datasets, many of which also have been explored in prior studies, are primarily used to demonstrate the advantages of applying TDA, as their underlying geometric structures provide valuable insights into the nature of the data. First, the classic MNIST dataset serves as an excellent example where prominent underlying geometric features (e.g., the number of holes, curvatures, etc.) can be effectively utilized for classification tasks. This dataset was also employed in [1] and [2], although the dataset was transformed into a point cloud/graph/mesh in [2]. The Br35H dataset, with its larger dimensions (112 × 112) compared to MNIST (28 × 28), features brain tumors that manifest as loops or connected components, making it particularly suitable for demonstrating ECC’s computational efficiency. Moreover, in our final revised paper, we shall conduct additional experiments on ORBIT5K point cloud data, a dataset that is often used as a benchmark in TDA and is also employed in [1] and [3].

10. [Ans to Q10] Yes, persistence diagrams for 60000 images of size 28x28 were calculated in 33 seconds on the Apple M2 cpu. We utilized the GUDHI package to compute PH, using a superlevel set filtration applied to a V-constructed cubical complex.

11. [Ans to Q11] We apologize for the error in our wording. Our intention was to convey that calculating PH for 112 x 112 images is ‘impractical’ or ‘inefficient’ due to computational burdens, rather than ‘infeasible.’ This has been corrected in the updated paper.

12. [Ans to Q12] ECC is considered a weaker topological invariant because it aggregates topological features across all homology dimensions, whereas PH provides separate topological summaries for each homology dimension. Intuitively, this makes ECC less stable than PH, as changes in the data can lead to more significant alterations in the aggregated information compared to the changes observed individually in each homology dimension. This intuition aligns with the mathematical proof: ECC is bounded by the Wasserstein distance, which is less strict than the Bottleneck distance that bounds PH. Moreover, even if we don't aggregate on homology dimensions, ECC is something like a rank function; all the noisy homological features at the given flitration value $t$ are accumulated to the rank function values at that $t$. Adaptively reflecting homological features according to their importances is almost impossible. While in the persistence diagram, noisy homological features are presented as individual points, and one can adaptively reflect homological features according to their importances: for e.g., using $\ell_{\infty}$ type distance such as the bottleneck distance will disregard noisy homological features below threshold. This phenomenon is also partially due to that persistence diagrams have much more complicated geometrical structure that can accommodate complicated topological difference, compared to functional spaces.

We appreciate the great interest you have shown in our work, and address each of your questions below.

1. [Ans to Q1]: Thank you for pointing this out. We have addressed this in our revised paper, adding it as a footnote on page 3.

2. [Ans to Q2]: As the reviewer suggested, ECC with heights from all directions in Turner et al. has the full information of the shape. However, in the learning situation, having the full information is not directly transferred to the most useful learning in any sense (high accuracy, robust, etc). ECC with heights from all directions needs (with discretizing the directions) a high dimensional space to encode that information. Also, these encoded features are highly correlated, i.e., for e.g., when directions $v$ and $w$ are close, then corresponding ECC are very similar to each other as well. Hence in the setting where training data are not sufficient or moderate noise are present, ECC with all directions won't be appropriately trained due to its high dimensionality and correlation. Also, the height function from a direction is vulnerable to noise, in particular to outliers. With our approach, we can use a filtration that is robust to noise, for e.g., the distance to measure (DTM) filtration, and make the corresponding ECC robust to noise as well.

3. [Ans to Q3]: PLLay and DECT are most closely related works to ours, as they are the only prior works that address differentiability with respect to the layer input, enabling backpropagation through the layer. In contrast, PersLay introduces a framework for learning vectorizations of persistence diagrams within a deep learning context, but does not support backpropagation through the layer.

4. [Ans to Q4]: This is a good point. The compromise in connectivity information is discussed in the context of DECT [2], where the authors directly applied the height filtration to MNIST and ModelNet point clouds by simply counting the number of points above or below a hyperplane. While we find your suggested method plausible, it seems more aligned with applying the height filtration to a graph or mesh derived from a point cloud, rather than directly to the point cloud itself. Furthermore, the ECC from the non-vanilla Vietoris-Rips filtration as you suggested might be able to use the connectivity information fully, though it still suffers from noise, in particular to outliers, as we have explained in Ans to Q2.

5. [Ans to Q5]: Thank you for bringing this to our attention. We have addressed this in our revised paper to ensure clarity and avoid any potential confusion.

6. [Ans to Q6]: Yes, a naive and convenient way of setting the interval would be to assign $T_{min}$ and $T_{max}$ as the the minimum and maximum of the possible filtration values, respectively. Alternatively, one could select $[T_{min}, T_{max}]$ as a tighter interval within the range of possible filtration values, focusing on regions of the filtration that contain meaningful topological and geometrical information. Such an interval can be identified via hyperparameter search, or chosen intuitively by examining the ECC computed for some data. We have included a discussion on this topic in Appendix H of the revised manuscript.

7. [Ans to Q7] Thank you for this insightful question. As you pointed out, given the discretized interval, the vectorized ECCs are not able to capture cycles that are born and perished between $t_i$ and $t_{i+1}$. However, cycles with very short life spans are often insignificant (noisy) generators. Therefore, excessively dense discretization may not be beneficial, as including numerous small, noisy generators in the learning process can lead to increased variance. (please refer to the illustration provided in Figure 6 of the revised manuscript.) On the other hand, the discretization should also avoid being overly sparse, as this could compromise the ability to capture essential global features. Determining the optimal choice of discretizing bins is a non-trivial task, which we do not address in the current study. However, as demonstrated in our simulations, standard data-driven heuristics, such as cross-validation or a simple grid search, can be employed to select an appropriate value for $v$ that effectively balances the aforementioned trade-offs. Also importantly, avoiding excessively fine discretization is essential to prevent overly loose stability bounds in Proposition 5.1. We make this clear in our revised manuscript.