Defining and Measuring Disentanglement for non-Independent Factors of Variation

We would like to start by thanking you for your review. Next we try to address each of your questions and concerns. Next bullet points follow the same order as yours in Weaknesses section:

• Actually we do not propose a upper bound but an approximation. During the development of this work, our first goal was to propose a upper bound but we finally found a stronger metric, which was an approximation, that is the current one. We forgot to change some of the sentences in the demonstrations in the Appendices. Please see the new version. Thanks for noticing this point.
• This second point refers to different concerns related to the experiments section. We agree with the last one but we disagree with the others. We explain why below:
  1. "Multiple design choices of the authors are not motivated.". We would appreciate it if you could specify better what are the design choices that we do not motivate. The only design choice that we make is that of defining the datasets and we believe that they are sufficiently motivated. We would thank you if you could be more specific with this point.
  2. "The authors did not investigate their metrics's correlation with down-stream task utility.". Please, find Point 2 in Answer to Reviewer M4R5 II.
  3. "The authors never looked at actually learned representations, but only used simple linear (and in the first section, trigonometric) mappings.". The reason for this is simply that if we analyze some learned representations, we cannot know if these representations are actually disentangled. In fact, our metric's goal is to analyze if learned representations are disentangled. Thus, we cannot analyze how good the performance of our metric is through learned representations since we would be falling into a circulus in probando, which is a logical fallacy. Reviewer M4R5 also pointed out the inconvenience of using learned representations for our proposal.
  4. "The discussion the authors give is mostly limited to extreme/edge cases of their metric. The metrics overall behaviour is not discussed.". See new Experiment 5.2.

We answer next the Questions following the same order as yours:

• Please find the answer in the first point of the previous list.
• For the case of minimality, the reason for choosing the maximum is that we assume that the factor "associated" to a variable $j$ corresponds to the $\arg\max_{i} m_{ij}$. Thus, we want to know how much information of $z_j$ contains its "associated" factor variation $y_i$. Same reasoning can be applied to sufficiency.
• See "Answer to All Reviewers".
• See "Answer to All Reviewers".
• See "Answer to All Reviewers".
• See "Answer to All Reviewers".

We hope that we have clarified your questions and strengthen our weaknesses. If so and you believe that current version of the work deserves an increase in the score, we would appreciate it if you make that increase effective. Otherwise, we will be willing to continue with the discussion.

Thank you for your quick response. We answer below your comments.

• In the new experiment, we do not contemplate the case in which $0.5 < \alpha$ because it can become hard to interpret. If $0.5 < \alpha$, then there is not a factor of variation that is always more important than others, conclusions are harder to extract.
• We have slightly modified the experiment 5.2 and we do not find this phenomenon now. We have not found any reason why this was happening in the experiment in the previous version of our manuscript.
• Figures 13, 14 and 15 are now modified (in fact, paper no longer has figures 14 and 15). In explanation of new section 5.2 we give the reason why Sufficiency takes the values that it takes and the reason why this is a desirable behaviour.

I still believe, that this paper would benefit from showcasing (not validating) the metric on actual learned representations. Showcasing your metric on actual learned representations of popular disentanglement datasets and analysing the dimensions identified by your metric (e.g. visually) would be convincing and showcase actual utility for the community.

We do not think that obtaining representations from data and just showing the values of our metrics for these representations helps to make the paper more convincing. However, maybe the next experiment can help to understand the necessity of our proposal: Train a variation of a VAE that tries to learn disentangled representations for a dataset with correlated factors of variation (e.g. one of those used in Experiment 5.2). Then, show that we can generate data in a controlled way (and thus, the representations are presumably disentangled) and that, despite of this, other metrics in the literature indicate that the representations are not disentangled. If you believe that this experiment can help to showcasing our metrics, we can include it in an appendix in the final version of the paper. However we do not believe that this must be in the main text since, as explained in Point 2 in Answer to Reviewer M4R5 II, we think that measuring disentanglement performance through the performance in a downstream task is not methodologically correct.

I am still sceptic of the authors approach of using the maximum in their metric. Papers such as [1] have showcased, that often multiple dimensions have very similar contributions to one generative factor. Also in the synthetic experiments the authors propose themselves, the contributions of multiple dimensions can be very similar. The "winner takes it all"-approach makes a metric behave more erratic, tiny differences in how much a dimension encodes can then alter the metric dramatically.

This is exactly the goal of our paper. We want to propose a method that allows to different dimensions of the representation to have some information about a factor of variation $y_i$. The point is that in some of the dimensions this information must come through another factor of variation different than $y_i$ (but that is correlated with $y_i$). This is the concept of conditional independence, and this is why we propose using conditional mutual information rather than mutual information to measure disentanglement. In section 3 we argue why this is desirable and in section 4 we demonstrate that our metrics are actually (almost) measuring this.

We would like to start by thanking you for your review. We answer each of your concerns and questions in the Weaknesses section following the same structure that you propose:

Clarification on minimality and sufficiency

Your interpretation is correct. Although the concepts of sufficient and minimal representation have been used in multiple works to refer to one representation and one task, nothing prevents one from doing it for more than one. Actually, as explained in the second paragraph os section 3, the $z_j \in \\pmb{z}$ can be seen as representations. Equivalently, one can see each factor of variation as tasks (actually the word task is just a generalization for something that you want to learn and potentially solve). Thus, we could say that we want to perfectly "solve" only one factor $y_i$ of variation with only one representation $z_j$. In section 3 we connect [1,2] with the four properties and in section 4.1 we connect the four properties with minimal and sufficient representations. We don't know if you consider this a sufficient connection, but for us it is the most clear way of connecting our paper to the two previous.

Beginning of Sec. 5

You are right. We explain our solution for both issues. Please let us know if you find them proper.

1. Since the metrics are introduced in section 4.3 (the title of the section starts with the word Metrics) we make this clarification in this section. Concretely, after defining $\bar{m}$ and $\bar{s}$, which are the metrics that we propose.
2. Now we use italic for metrics and roman for properties.

Experiments

We try to address each of your specific concerns below:

1. This is an interesting question: "Should a metric have an intrinsic meaning or should it be useful only to compare methods?". Although, we do not have clear answer for this, we believe that the first option is never worse than the second, since the first option implies the second one, while the converse is not necessarily true. In fact, many of the most widely used metrics in machine learning are designed to be interpretable (e.g. accuracy, f1-score, BLEU, MAP). Our metrics have a clear interpretation by definition (and, as a consequence, they also allow to compare representations). For example, minimality captures the proportion of information that a variable $z_j$ contains about a factor $y_i$ with respect to the the input $x$. However, other metrics in the literature (e.g. DCI-D) allow to compare representations but they do lack of an easy interpretation. We could add a rank correlation matrix between different metrics in some of the datasets but we do not believe that this would help to draw any specific conclusion on our metrics. However, we are open to discuss this point further, since we find it interesting and we do not have a clear answer to it.

2. This is also an interesting question. However, we believe that comparing the value of a disentanglement metric with the performance in a subset of downstream tasks is not convenient in a paper proposing a definition metric for disentanglement for several reasons, which we list below:

   • We believe that defining a way of measuring disentanglement has an intrinsic value. Giving the correlation of disentanglement metric and downstream performance could mask the actual intention of the paper.
   • It is well known that disentangled representations translate into a better performance in downstream tasks. Thus, if out metric correctly measure disentanglement, this should translate into a good correlation between our metric and downstream tasks.
   • Most importantly and independently of the aforementioned, we believe that measuring the performance in downstream tasks to asses how good a disentanglement metric is is methodologically incorrect. Imagine that we carry out these experiments and the results are successful (in the sense that a disentanglement metric perfectly correlates with downstream performance). Then, saying that this metric perfectly correlates with disentanglement is a logical fallacy, as we can see next: Let $p\equiv *high disentanglement*$, $q\equiv *high value of the disentanglement metric*$ and $r\equiv *good performance in downstream tasks*$. Thus, we have $p \Rightarrow r$ (since we assume that having highly disentangled representations implies a good performance in downstream tasks) and $q \Leftrightarrow r$ (since we assume that the experiments in downstream tasks are successful). Under this scenario we can infer $p \Rightarrow q$ but not $q \Rightarrow p$, i.e., we know that our metric will be high when the disentanglement is high, but our metric can be high even though the disentanglement is low. This derives from the fact that, to the best of our knowledge, it has not be proven that disentanglement is a necessary condition to have good performance in any set of downstream tasks. Thus, measuring performance in downstream tasks does not allow to obtain a bidirectional conclusion. In fact, to the best of our knowledge, it is hard to find a paper proposing a definition and metric for disentanglement apart from that of DCI-ES that perform some experiments in downstream tasks. We are also open to discuss further this point since it is also of interest of us.

3. See "Answer to All Reviewers".

4. We use a wide range of mixing degrees (which varies with $\alpha$, $\beta$, $\gamma$, $\delta$ and $\sigma$) along the different experiments. With respect to the ways of mixing, especially if compare the experiments we do with those of DCI-ES paper, we have the next:

   • We use their Noisy Labels with a wider range of noise values, and their Linearly-mixed labels with a wide variety of $W$ matrices.
   • With respect to their Raw data (pixels) and Others, we believe that it does not make sense to analyze this case in our paper, since we do not know the "ground-truth" level of disentanglement in these cases. Thus, we cannot know if the results that we would obtain make sense or not.

5. You are right: actual images are never used, but only the factors of variation. Generating randomly the factors is something we do in section 5.1. We found it interesting to use some correlated factors of variation from existing papers, which is what we do in section 5.2.

Minors

• It should be "those described in section 2". Thanks for noticing.
• We extended the sentence.
• We made some modifications to fix this issue. Please see last paragraph of section 4.3 and Algorithms 1 and 2.
• This is an interesting point. Although in our paper it is a bit oversimplified to avoid misunderstandings, our opinion comparing the metrics that you mention are the next:
  • Explicitness ($E$) refers to the existence of a simple (e.g., linear) mapping from the representation to the value of a factor while Informativeness ($I$) refers to the amount of information that a representation captures about the underlying. That is, $E \Rightarrow I$, but $I \centernot\Rightarrow E$. Thus, theoretically, they are different. However in practice, $I$ is calculated also through the use of a mapping from the representation to a factor (due to the intractability of the integrals that calculating mutual information involves). Thus, the only practical difference is the type of mapping used to go from the representation to the factor. In fact, both can be seen as ways of Predictive $\mathcal{V}$-information [3] with different Predictive Family $\mathcal{V}$. Concretely, given the Predictive Family used to calculate the Explicitness $\mathcal{V}_E$ and the Predictive Family used to calculate the Informativeness $\mathcal{V}_I$, we have that $\mathcal{V}_E \subset \mathcal{V}_I$, and this is why $E \Rightarrow I$, but $I \centernot\Rightarrow E$.
  • As far as we understand them, Compactness and Completeness refer to broadly the same idea: The degree to which each underlying factor is captured by one (or a few) representation variable and there is a metric that has been specifically designed to measure Completeness but not to measure Compactness.

Sorry for the long answer, but some of your questions refer to ideas that we find interesting to discuss about. We hope that we have clarified your questions and strengthen our weaknesses. If so and you believe that current version of the work deserves an increase in the score, we would appreciate it if you make that increase effective. Otherwise, we will be willing to continue with the discussion.

[1] Eastwood, C., & Williams, C. K. (2018, May). A framework for the quantitative evaluation of disentangled representations. In 6th International Conference on Learning Representations.

[2] Ridgeway, K., & Mozer, M. C. (2018). Learning deep disentangled embeddings with the f-statistic loss. Advances in neural information processing systems, 31.

[3] Xu, Y., Zhao, S., Song, J., Stewart, R., & Ermon, S. (2020). A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689.

Additional comments

I agree that, in order to thoroughly check that these metrics do what expected in highly controlled settings, it would not be particularly helpful to run experiments on learned representations. However, similarly to k1fY's point, I think using learned representations, even on toy image data, is right now a major missing piece in this work -- such experiments would be useful for showcasing the behaviour of these metrics in scenarios that are slightly closer to real applications (which in a way has always been one of the core motivations for disentangled representations, see e.g. Bengio et al. (2013)).

There's also a related point: Träuble et al. and Roth et al., both cited in the submission, use in fact DCI-C as their primary metric without apparent practical issues. E.g. in Träuble et al. the weakly-supervised approach by Locatello et al. [4] allows to learn disentangled representations even on correlated data, and the DCI-D metric is quite high in those cases, and actually from the violin plots you can see it's often very close to 1. Does this somehow depend on the fact that the representations were learned with neural networks, or on the weak supervision in Träuble et al.? Or maybe on the way and/or degree that the factors were entangled in the representations, or on how precisely they were correlated in the data, or on an interplay of the two?

I guess my point is the following. I fully agree with the authors' theoretical arguments, which I find compelling and significant. However, while I am not doubting their empirical results, I think they address only a very narrow setting, which limits the depth of understanding of the proposed metric.

Finally just another note -- I see the point that investigating the correlation with downstream performance is orthogonal, but this would be highly relevant for the community and the authors are missing an opportunity to have an additional, more pragmatic selling point.

References

[1] Van Steenkiste, et al. "Are disentangled representations helpful for abstract visual reasoning?", NeurIPS 2019

[2] Dittadi, et al. "On the Transfer of Disentangled Representations in Realistic Settings", ICLR 2021

[3] Träuble, et al. "The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agents", ICLR 2022

[4] Locatello et al. "Weakly-supervised disentanglement without compromises", ICML 2020

Thanks for the extensive and interesting rebuttal and for the clarifications. I'm also glad to hear the authors found some questions interesting to discuss! I still have a few more comments on some points.

Answers about experiments

1. That's a good point and I agree that your metric measures something sensible and well-defined. However, I disagree on your point about "clear interpretation". You write "minimality captures the proportion of information that a variable $z_j$ contains about a factor $y_i$ with respect to the the input $x$", and this is imho not intuitively interpretable. E.g. if the metric is equal to 0.76 I'm not sure what to make of it; I know that if another one is 0.84, the latter is supposedly better than the former, but the absolute values are not interpretable. I believe that, in principle, even if they are not interpretable, they are meaningful since they are theoretically justified. They are an estimate of a mathematical quantity that is sensible to want to quantify. But I would argue against saying that e.g. DCI-D is not meaningful. In the uncorrelated case, it is not surprising that some standard metrics may still work well -- that does not undermine this work at all, and overclaiming does it a disservice. If e.g. DCI-D is (almost) a monotonic function of your metric, then your metric in practice does not buy you anything. You can argue that it measures something more sensible, but that's about it. On the other hand, the story is completely different on the dependent case or with nuisance factors, which are the 2 main motivations of this work. I would focus the claim on these cases.

2. • "It is well known that disentangled representations translate into a better performance in downstream tasks": I disagree. Disentangled representations are often presented as something desirable, but it's clear by now that they are not always helpful (except for specific cases like interpretability or controllable generation) -- e.g., it depends on the downstream task, downstream model, whether there's a distribution shift at test-time, the nature of such shift, etc. See for example: the 2 papers by Montero et al. that you even already cite; "Challenging common assumptions" which you also already cite, where there's limited evidence of usefulness; [1] where on a harder downstream task there is better sample efficiency but otherwise there's no clear advantage; [2] where it is highlighted how strongly things depend on the downstream model, and for a simple MLP (as opposed to boosted trees used in previous papers) there is a mild advantage only in terms of OOD generalization; [3] where on a RL task disentanglement doesn't seem to help even OOD.
   • In addition, you write "Thus, if our metric correctly measure disentanglement, this should translate into a good correlation between our metric and downstream tasks." Previous works have investigated the empirical usefulness of disentanglement through specific metrics. If you expect this correlation to hold, your statement implies that you are measuring disentanglement in a similar way to previous works, therefore making your work less useful in practice (e.g. another metric would rank representations in a similar order to the one yours would). Again, you have the correlated and nuisance cases working for you, so I would not try to claim too much about the standard uncorrelated case. Some previous metrics already work pretty well there, and that is completely fine.
   • I agree. My point about usefulness of the metric was in fact orthogonal: if a metric does not necessarily measure disentanglement perfectly (or rather, we cannot know that for certain) but perfectly correlates with some downstream metric we care about (e.g. OOD accuracy on a few different downstream tasks), then it is in any case useful in practice, and the community should know. But again, I agree that it does not necessarily tell us about disentanglement. (In fact, I would even say that it's not necessarily true that $p$ implies $r$, but that is a side note.)

3. Thanks, the updated results make sense.

4. You are right, of course. What I meant to write was in fact to have a wide range of nonlinearly mixed representations e.g. by randomly-initialized invertible non-linear transformations (e.g. a composition of a few normalizing flows). I would find it interesting because it would get a bit closer to realistic scenarios while still in a highly controlled setting. However, I still consider it a relatively minor point and I'm not giving it much weight.

5. Ok, but it is still quite confusing that the title of Sec. 5.2 is "literature datasets" and the datasets in question are image dataset, but at the same time the images are never used. As far as I can tell, the same exact experiments could be run without even mentioning these datasets. Maybe I'm missing something, but that is at the very least unclear.

We would like to start by thanking you for your review. Next we try propose solutions to strengthen the weaknesses that you pointed out. Each bullet point corresponds to each of your two paragraphs.

• CelebA presents a very low level of correlation between the factors of variation. Thus, other metrics perform reasonably well with this. We have extended Experiment 5.2 to make it more "realistic" and to include different levels of disentanglement. Please see "Answer to All Reviewers".
• The authors are conscious of the connection between minimal-sufficient representations and the Information Bottleneck (IB) and we appreciate the importance of this concept in representation learning. Thus, we have added a paragraph about the IB in section 2, since we agree that this will help to give a better understanding of our approach and to better connect it to other works. However, as far as we know, ours is the first approach connecting the IB with disentangled representation learning. We would thank you if you could provide us some other works doing so in case you are aware of some. \end{itemize}

Next, we try to answer your concrete questions:

1. We believe that the only limitation is how $f_{ij}$ are chosen. If these regressors are too simple, maybe it could be hard for them to correctly discover the connections between the factors of variation $y_i$ and the representations $z_j$ even if there is a high mutual information between them. This point is briefly commented now in the last sentence of section 4.3. Apart from this, we believe that, assuming that definitions of section 3 are considered reasonable, our metrics should lack of other limitations than the one corresponding to the capacity of the regressors. Concretely, if we assume that definition Section 3 is correct, we have the next: (i) in section 4.1 we demonstrate that this definition is equivalent to having minimal-sufficient representations; (ii) and in section 4.3 we simply refer to the definition to minimal-sufficient representations to formulate our metrics.
2. As mentioned in the previous answer, we believe that the only limitation of our metrics is the way the regressors are chosen. However, as explained in section 2, the most accepted metrics in the literature [1,2] also make use of the regressors. Thus, they present the same limitation as ours.
3. First, if we study some of the most used metrics in the literature (DCI or modularity, compactness and explicitness (MCE)) we find it hard to modify them to relax the independence assumption. We believe that they need of too many changes to be considered a modification of the metrics and that they should be considered, if anything, a new metric. Even if our previous affirmation is wrong and we could straightforwardly modify DCI or MCE to make them robust to non-independent factors of variation, the previous metrics would not measure nuisances-invariance, which is a pivotal property that is overlooked in all the methods reviewed in section 2 and that our metric captures.

We hope that we have clarified your questions and strengthen our weaknesses. If so and you believe that current version of the work deserves an increase in the score, we would appreciate it if you make that increase effective. Otherwise, we will be willing to continue with the discussion.

[1] Eastwood, C., & Williams, C. K. (2018, May). A framework for the quantitative evaluation of disentangled representations. In 6th International Conference on Learning Representations.

[2] Ridgeway, K., & Mozer, M. C. (2018). Learning deep disentangled embeddings with the f-statistic loss. Advances in neural information processing systems, 31.

Thank you again for your feedback. We answer below your last comments.

I'm a bit surprised by this; generally, for the less 'extensive' datasets to explore disentanglement in, CelebA is known to be the case where one cannot assume independence between factors (e.g. beard vs gender, etc). It would be useful to get a handle on how much correlation would be required for this method to be useful then?

You are right. There is some correlation between factors in CelebA. However, what we meant is that, since many of the factors are almost uncorrelated to each other, the total correlation is very low. This implies that other metrics approximately perform reasonably well. Since we agree in that analyzing the case with more factors of variation is interesting, we added a similar experiment to that of Section 5.2 but for CelebA. Thus, we can analyze what happens when we have more factors of variation and strong correlations between factors. You can find the results and description in Appendices D and E, respectively.

I'm not quite sure such a blanket statement can be made about IB and representation learning, particularly on the back of prior approaches to leverage IB (and the rate-distortion principle more generally) to effect meaningful representations.

You are right. We weren't aware of those papers. The paragraph related to the IB in section 2 has been appropriately modified. The main difference between our works and the previous is that ours formally define a definition of disentanglement, as stated now in the paper.

We hope that these two last changes have helped to improve the level of convincingness that you believe our work has. You pointed out two weaknesses in the original review and we have tried to do our best to fix them:

1. The absence of experimentation with more realistic datasets, such as CelebA. We performed some experiments with the original CelebA and adding some extra correlations between factors of variation of this.
2. The absence of mention and connection of our work to the Information Bottleneck in section 2. We addressed this: Now we introduce the concept of IB and we connect it to that of disentangled representation learning. We specify which is the main novelty of our work with the other works connecting IB and disentangled representation learning. If you are still unconvinced, we would appreciate if you could be more precise on what you think that our work is missing to result more convincing.

Right, its reasonable that there is some constraint on the complexity of the regressors, but what is not clear is that the sensitivity to what type of regressor used is the same in these prior approaches?

We would include in the final version some figures (similar to those in sections 5.1 and 5.2) comparing different regressors for our metrics. This would help to show the robustness of them.

We would like to start by thanking you for your review. Next we try to address each of your questions and concerns. Next bullet points follow the same order as yours in Weaknesses section:

• We consider that the factors of variation can be always completely defined by $x$. As it has been noted in identifiability literature, if the mixing function that generates $x$ from the factors of variation $y$ and nuisances $n$ is not invertible, then the problem is ill-posed (see Appendix A.5 in [1]). The fact that the this mixing function is considered to be invertible implies that all the information of all the factors of variation of $y$ is present in $x$. Thus, the best and worst possible values of minimality/sufficiency are always 1 and 0, respectively.
• It is true that the ground-truth values of $y$ need to be known in order to calculate the minimality/sufficiency, but as far as we know, this is a "problem" that all the metrics in the literature (those described in section 2) suffer. We could think of a scenario in which there is a subset $y'^{(k)} \subseteq y^{(k)}$ whose ground-truth value is known for a given input $x^{(k)}$, similarly to [2]. In this case, line 4 of Algorithm 1 and line 3 of Algorithm 2 should not be $i=1$ to $n$, but $i \in y'^{(k)}$. We believe, however that maybe this can be more confusing for the reader and we assume that all the ground-truth values are known, as other metrics do.
• The fact that the metrics are normalized is intentional and does not suppose any loose of robustness (in fact, the lack of robustness would come from having unnormalized quantities). We detail this next: Since $z_j$ is a representation of $x$, we know that $H(z_j|x)=0$ and, thus $H(z_j)=I(z_j;x)$. This means that in both scenarios that you propose $y_i$ "explains" a $10\%$ of $z_j$ (more correctly, contains a $10\%$ of $z_j$ of the information of $y_i$). Thus, the interpretation of minimality is the proportion of information of $z_j$ that $y_i$ contains with respect to $x$ in all the cases, no matter how entropic $z_j$ is. This is because some variables can be more entropic than others, but minimality should be robust to this. Equivalent reasoning can be applied to Sufficiency. From a theoretical point of view, we should not find any problem when having more factors and the time complexity is not higher than that of other algorithms in the literature.
• Lowercase notation is used in all the paper now.
• Typos have been solved. Thanks for noticing.

We hope that we have clarified your questions and strengthen our weaknesses. If so and you believe that current version of the work deserves an increase in the score, we would appreciate it if you make that increase effective. Otherwise, we will be willing to continue with the discussion.

[1] Lachapelle, S., Rodriguez, P., Sharma, Y., Everett, K. E., Le Priol, R., Lacoste, A., & Lacoste-Julien, S. (2022, June). Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In Conference on Causal Learning and Reasoning (pp. 428-484). PMLR.

[2] Locatello, F., Abbati, G., Rainforth, T., Bauer, S., Schölkopf, B., & Bachem, O. (2019). On the fairness of disentangled representations. Advances in neural information processing systems, 32.

Thank you for the clarifications. Below are my further comments regarding the first point.

We consider that the factors of variation can be always completely defined by $x$.

Yes, I understand that. However, I don't think the non-invertible setting is ill-posed, as many real-world problems are indeed not invertible. Appendix A.5 in [1] does not say the problem is ill-posed either. My point is that one may not know a priori how invertible a given problem is, and if it is not invertible, then proposed metrics may not reflect how good a representation really is. I agree with Reviewer M4R5 that properly ranking different representations may be a more important utility of the metrics here. On the other hand, the absolute score is only useful when the maximum possible score for the problem is known (e.g., about 100% for metrics like accuracy on most datasets).

[1] Träuble, F., Creager, E., Kilbertus, N., Locatello, F., Dittadi, A., Goyal, A., ... & Bauer, S. (2021, July). On disentangled representations learned from correlated data. In International conference on machine learning (pp. 10401-10412). PMLR.

[2] Roth, K., Ibrahim, M., Akata, Z., Vincent, P., & Bouchacourt, D. (2022). Disentanglement of correlated factors via hausdorff factorized support. arXiv preprint arXiv:2210.07347.

[3] Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., & Bachem, O. (2019, May). Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning (pp. 4114-4124). PMLR.

[4] Ng, A. Y. (2004, July). Feature selection, L 1 vs. L 2 regularization, and rotational invariance. In Proceedings of the twenty-first international conference on Machine learning (p. 78).

Fig 5.a. Uncorrelated

Fig 5.b. 1 Pair

Fig 5.c. 2 Pairs

Fig 5.d. 3 Pairs

Fig 5.e. Confounded

Fig 6.a. Uncorrelated

Fig 6.b. 1 Pair

Fig 6.b. 2 Pairs

Fig 6.c. 3 Pairs

Fig 6.a. Confounded