Deep Regression Representation Learning with Topology

Main Concerns

1. Clarity

   1. The underlying assumptions used to prove the statements in Section 3 are not fully clarified. Theorem 1 implicitly restricts the considered representations to deterministic functions of $\bf x$. Previous literature [1] has shown the benefit of using stochastic encoders, for which the result from Theorem 1 is not applicable.
   2. Theorem 3 assumes a uniform (conditional) distribution on the manifold $\mathcal{M}_i$ but the paper does not elaborate on under which conditions this assumption is reasonable.
   3. Section 4 is extremely difficult to follow because of the large number of definitions. Additional intuition on the roles of the terms in $\mathcal{L}'_d$, $\mathcal{L}_d$, $\mathcal{L}_t$ would greatly improve the readability.

2. Soundess

   1. The definition of the optimal representation and the statement in proposition 2 seem not to be applicable to continuous $\bf z$ and $\bf y$ without further clarifications. The value of the (differential) entropies $\mathcal{H}({\bf Y}|{\bf Z})$ and $\mathcal{H}({\bf Z}|{\bf Y})$ approach $-\infty$ whenever there exists an invertible mapping ${\bf y}=f({\bf z})$. As a result, I believe the proof for proposition 2 needs to be revised.
   2. The existence of a homeomorphic mapping between $\bf z$ and $\bf y$ and definition 1 rely on the existence of an underlying mapping ${\bf y} = f({\bf x})$. In other words, this assumption seems to restrict the setting to targets $\bf y$ that can be fully determined from $\bf x$. This aspect is not directly discussed in the main text and the real-world experiments consider distributions in which $p({\bf y}|{\bf x})$ has non-negligible aleatoric uncertainty (super-resolution, depth-estimation, age estimation).

3. Experiments

   1. No measure of standard deviation is reported, which makes it difficult to assess the significance of the reported results.
   2. The authors mention that "several widely used regularizers like weight decay and dropout effectively reduce the last hidden layer's intrinsic dimension", but these simple baselines are not included in any comparison.

Minor Issues

4. Presentation
   1. The distribution $\mathcal{D}$ defined in theorem 2 seems to depend on the specific sample ${\bf y}_i$ but the index $i$ is dropped in the notation.
   2. Figure 2b is helpful in supporting section 4, but I was unable to parse the plot on the right side since there is no direct reference to it in the description.
   3. The differences between plots (b) to (e) in Figure 3 are difficult to relate to the description in the main text since the plots appear quite similar to each other.
   4. Section 5.2 introduces a large number of abbreviations without previous mentions.

References

[1] Alemi, Alexander A., et al. "Deep variational information bottleneck." arXiv preprint arXiv:1612.00410 (2016).

There are some points of the theoretical analysis that I do not understand and prevent me from giving a proper evaluation. The analysis hinges on theorem 1, but there are two aspects of it that I do not understand:

1. Minimising the difference is not generally equivalent to minimising the ratio, implying that the equivalence is a result of I being the mutual information. Can the author provide details on this equivalence? Is there a difference between I and \mathcal{I} in the proof of theorem 10?

2. Both Z and Y are deterministic functions of X. What does it imply for the entropy of Z conditioned on Y? Is it only meaningful when the mapping from X to Y is one-to-many? Does your theory suffer from the problems associated with having deterministic functions, as discussed in 'ON THE INFORMATION BOTTLENECK THEORY OF DEEP LEARNING' by Saxe et al, ICLR 2018?

Additionally,

-The accessibility of Section 4 is limited to readers with advanced knowledge in topology.

-The experimental evaluation is limited to a very simple architecture (two-layer network with 100 hidden units)

1. I have concerns about the IB principle. Neural networks are deterministic functions, and, thus H(Y|Z) = 0 always. H(Z|Y) > 0 when different Z map to the same Y. This can happen when different X map the the same Y, because for real large networks different input objects X have different embeddings Z. So, some H(Z|Y)>0 depends only on the dataset itself, not the network for realistic scenarios.

2. $min_Z \left( I(Z,X) - \beta I(Z,Y) \right)$ and $min_Z \frac{I(Z,X)}{ \beta I(Z,Y) }$ are different problems. Thus, Theorem 1 is wrong.

3. The regularizer in eq. 8 is the topological loss from Moot et. al, 2020. You should explicitly mention it.

4. Sometimes the narration is clumsy, ex.: "Figure 1(a) provides a t-SNE visualization of the 100-dimensional feature space with a ’Mammoth’ shape target space." Here I don't understand what does 100-d space mean and what is "Mammoth target space". Also, I think for your case visualization with t-SNE is not a good choice since t-SNE often tears a manifold apart. Try to use PCA.

minor:

1. Caption to Figure 3: model' - typo
2. In Section 3, you write "Consider a dataset S = {xi, yi} with N samples xi , which typically is an image (xi \in Rdx1×dx2 )" color channels of image are missing.

Writing correctness and clarity: There were a number of issues with the writing that made it more challenging to read the work than it should have been. For instance, typos such as:

1. In the introduction, “The homeomorphic between two…” $\mapsto$ “The homeomorphism between two…”.
2. In the introduction, “…and in the topology view,…” $\mapsto$ “…and from the topological viewpoint,…”
3. In Section 5.1, “In contrast, naively lowering the intrinsic dimension ($+L’_{d}$ ) performs poorly and even worse than the baseline, i.e., Tours” $\mapsto$ "...i.e., torus.”

There were also a number of mathematical statements that were made that either didn’t make sense or were not true. For example:

1. The sentence “The continuity represents the $0$th Betti number in topology…” is incorrect. The $0$th Betti number captures the number of connected components in the topological space but has little to do with continuity (otherwise). For instance, a set of three unique (and discrete) points in $\mathbb{R}^3$ has $0$th Betti number $3$ and three disjoint spheres in $\mathbb{R}^3$ also has $0$th Betti number $3$.
2. Pretty much all topological data analysis is based on algebraic topology, so the word ‘algebraic’ in ‘algebraic topological data analysis’ is unnecessary.
3. “However, unlike classification, regression’s target space is naturally a topology space, rich in topology information crucial for the detailed task.” It isn’t clear why the target space of a classification task is less naturally a topological space. Perhaps what the paper means is that there is less topological structure in the discrete target space of classification tasks. Then again, if one is only using the $0$th homology groups (as this paper does), one is not capturing any higher dimensional structure anyway.
4. Proposition 2 concerns continuous maps between $\mathbf{Z}$ and $\mathbf{Y}$. What are $\mathbf{Z}$ and $\mathbf{Y}$? Continuous maps only make sense when one has a defined topology on the domain and target space, this doesn’t seem to currently exist in the work.

There were also a few cases where notation was used that was never defined.

1. What is $\mathcal{Y}$ in Theorem 2?
2. If $\mathbf{X}$, $\mathbf{Z}$, and $\mathbf{Y}$ are going to be used in proofs, it should be stated what they are.

Finally, the reviewer had a hard time understanding some of the figures

1. Figure 1(b) is used to illustrate a point about intrinsic dimension in the introduction. The reviewer had trouble understanding what the task is here and what the reader is meant to see in this scatter plot. More context should probably be added.

Use of the word ‘topological’ when only $H_0$ is used: In the cases considered in this paper, the only topological property that $H_0$ measures is the number of connected components of a space (it has been shown that persistent homology also captures geometric information like curvature, so such features may also be leaking into regularization). As such, it captures only a small fraction of the total topological characterization of the space (particularly for high-dimensional spaces). It is this reviewer’s opinion that this fact should be more explicitly stated than what is currently found on the bottom of page 5. For instance, it would be more accurate to call this ‘connectedness regularization’. It is probably not necessary to actually make this shift, but it does highlight that what is being advertised in much of the work is only marginally captured by the actual algorithms. Also, the paper makes a point of noting that regression tasks have richer topology than classification tasks, but if one is only using $H_0$, the target spaces are effectively the same.

Concern about where improvements are coming from: One interpretation of the proposed method is that by constraining topology and intrinsic dimension (even on $\mathbf{Z}$) based on ground truth, one is imposing built-in priors to the regression model. With this extra information, it is not surprising that the new model achieves better performance. One way to test this would be to impose the same regularization on model outputs in $\mathbf{Y}$ rather than $\mathbf{Z}$. Are the same improvements seen? I am not sure that the results are less interesting, but the connection to IB might be less compelling.

Overall, I have some concerns regarding Theorems 1, 2 and 3, especially Theorem 2.

1. I have several concerns regarding the new generalization error bound in Eq.~(2).

1.1 how this bound is connected to the new bound in [1], which emphasized the role of I(X;Z|Y) in classification tasks.

[1] Kawaguchi, Kenji, et al. "How Does Information Bottleneck Help Deep Learning?." arXiv preprint arXiv:2305.18887 (2023).

1.2 Is the bound tighter or not? or does H(Z|Y) a good indicator on the generalization error in regression problems? Can you validate this point?

1.3 In fact, the Theorem 2 is a bit unclear, especially the description on Q. Are there some properties that Q should have?

1.4 It is unclear how to derive from Eq.(28) to Eq.(29). I understand that $(E(|z-\bar{z}|_2))^2 \leq E |z-\bar{z}|_2^2$. However, I do not understand why $E |z-\bar{z}|_2^2$ can be further bounded by $Q(H(Z|Y))$. Does $Q$ exist for general case? In fact, authors only illustrated that $Q$ exists for Gaussian case and uniform case. I am not sure if the derivation from Eq.(28) to Eq.(29) holds for general case.

2. Regarding Theorem 1, it only holds for a deterministic mapping function from $X$ to $Z$, since you assume that $H(Z|X)=0$? This should be emphasized in the main paper, since majority of deep IB approaches use stochastic representations and $H(Z|X)\neq 0$. Actually, this also decreases the applicability of Theorem 1.

3. Theorem 3 is actually a straightforward result of [Ghosh & Motani (2023), Proposition 1]. This should also be emphasized in the main paper.

4. Apart from the above concerns, I also feel the two regularization terms are not novel. In fact, the connection between entropy and intrinsic dimension has been carefully discussed in [Ghosh & Motani (2023)]. The way that authors evaluate the intrinsic dimenion seems to be directly from Birdal et al. (2021). As for another regularization term on topology perserving, it also shares rather similar to that in (Moor et al., 2020).

5. Finally, regarding the experiments. It would be much helpful if authors can compare their method to other prevalent deep IB approaches. For example, nonlinear information bottleneck also considers a regression setup. Another example is the convex information bottleneck [2].

[2] Rodríguez Gálvez, Borja, Ragnar Thobaben, and Mikael Skoglund. "The convex information bottleneck lagrangian." Entropy 22.

1. The proposed "intrinsic dimension lowering" loss term $\mathcal{L}_d$ actually disturbs the intrinsic dimension in an unclear way, it can both increase and decrease it, since, for a given data representation $Z$, the term $\mathcal{L}_d$ involves the ratio of logarithms $\log E(Z_n)/ \log E(Y_n)$ where $Y$ is another data representation. There is no $\log E_n(Y)$, $Y-$dependent part, in the formula for intrinsic dimension of $Z$, see eg arXiv:2306.04723 or J. M. Steele, Growth rates of euclidean minimal spanning trees with power weighted edges, The Annals of Probability, 16(4):1767–1787, 1988. So the paper narrative concerning the lowering of intrinsic dimension is seemingly in contradiction with the reported experiments setup.

2. Reproducibility check concerning the reported experiments in the manuscript (Section 5) could not be performed. No source code was made available during the review phase.

3. Theoretical results reported in Section 3 seem to have strong intersection with previously published papers, in particular, Theorem 3 from Section 3 seems to be similar to Proposition 1 from [Ghosh & Motani (2023)], cf Appendix A.3.

4. The definition of the intrinsic dimension employed in Section 3, which is the same as the definition in [Ghosh & Motani (2023)], is completely different from the definition of intrinsic dimension via the 0-th persistent homology, employed in Section 4. So, a priori, there might be no connection between the two quantities.

5. Training details are lacking in the description of the experiments reported in Table 1, Table 2, Table 3, Table 4. In particular, how many iterations/epochs were used, what were the stopping criteria? It might be that the experiments, if they can be reproduced, cf above, could be explained by overfitting of the baselines, since it seems that the standard regularizers, such as weight decay, were not used.

6. Standard deviations are not reported in Table 2, Table 3, Table 4.

7. Only a single baseline method is employed for comparison in each case in Table 2, Table 3, Table 4.

8. The loss $\mathcal{L}_t$ employed "to preserve topology", has been recently shown to have the following drawbacks: firstly, this loss is not continuous, secondly, moreover, diminishing $\mathcal{L}_t$ can lead to bigger difference in topology, cf arxiv2302.00136, Appendix J, Figure 10.

9. Related work section does not mention many relevant papers, eg arXiv:2106.04024, arXiv:2201.00058, arXiv:2306.04723, J. M. Steele, Growth rates of euclidean minimal spanning trees with power weighted edges, The Annals of Probability, 16(4):1767–1787, 1988.

10. The writing could be improved, there are some vague statements and not very clearly defined notions.

Below are some specific remarks:

page 2 : "intrinsic dimension of the feature space" - what is this ? From what follows a reader can guess that seemingly it is what can be called intrinsic dimension of the dataset points in the feature space, is it?

page 2 : Figure 1a - The right picture with supposedly "topology similar" has clearly too many clusters, eg a separate leg cluster, and, on the contrary, it does not preserve topology. The preserving topology representations exist in the literature for this mammooth dataset and they look completely different.

page 2 : Figure 1b "Lower the intrinsic dimension" picture - Is the paper saying that a good representation of the 3d mammoth for regression task should be 1dimensional? First of all, the 3d object smashed into 1d space does not seem to be a good representation for the vast majority of tasks including many other regression tasks. Secondly, such a smashing into 1d space representation would depend heavily on the regression task, which is also usually considered to be bad for a data representation.

page 2: "The homeomorphic between two spaces" - should it be homeomorphism ?

page 2: " t-SNE visualization of the 100-dimensional feature space" - the 100d feature space is described only in Section 5.1, several pages down. It would be better to mention it here on page 3, otherwise it is impossible to understand what are the 100 dimensions mentioned here.

page 2: "we are the first to explore topology in the context of regression representation learning"- actually, the autoencoders learn the regression task of reproducing the coordinates of the data points, so no, the topology has been heavily explored in previous works in the context of regression representation learning.

page 3: "The intrinsic dimension of the last hidden layer" - what is this ? From other contexts a reader can probably guess that, seemingly, it is what can be called intrinsic dimension of the dataset points in the last hidden layer, is it?

page 3: "1-dimensional (Trofimov et al 2023) topologically relevant distances" - actually the method from (Trofimov et al 2023) preserves both 0-dimensional and 1-dimensional topological features, and even arbitrary $k-$dimensional features.

page 3: "However, unlike classification," - as already mentioned, the autoencoders learn the regression task of reproducing the coordinates of dataset points.

page 4: "$Dim_{ID} M_i$ is the intrinsic dimension of the manifold"(corresponding to the distribution) - which definition of intrinsic dimension is used here?

page 5: "is a homomorphism between ... and" - is a homomorphism from... to ...

page 5: "that the set of points S sampled from" - that the set of points S is sampled from ?