Correcting Flaws in Common Disentanglement Metrics

The concepts presented can be seen as incremental improvements over existing ideas. The two main idea of this paper, neuron-factor alignment and the difference between individual vs collective informativeness, are, to our knowledge, completely novel. We do not know of any existing works to explore these ideas.

How do the proposed metrics compare with other existing metrics in terms of computational efficiency and scalability? SNC is fast, <1 minute for the entire dataset. NK is slower, around 20minutes for the full dataset. This is still faster than DCI, which can take up to one hour.

Discussion of MIG-sup. Thank for this reference, we will include it in the discussion of single neurons representing multiple factors. It is indeed a solution to this problem, but we would argue it is an imperfect solution. In the comparison between the two toy models in Section 3.1, M1 would get a higher MIG score but a lower MIG-sup, and the sum MIG+MIG-sup would be the same for both models. For our metrics, however, the sum SNC+NK would be lower, which we have argued to be more correct. There is also the problem that it is not clear how to extend MIG and MIG-sup to deal with P2, because extending MI to multiple variables becomes tricky both theoretically and computationally.

Modularity in [2] takes a similar solution to this problem, similar also to DCI, though like DCI, we do not claim that this problem occurs in [2] because it doesn't operate by explicitly aligning factors to neurons, as MIG does. Also, all three of these approaches, DCI, MIG and [2], penalize a neuron representing multiple factors by adding another metric. Our work instead employs the cleaner solution of forcing the alignment to be injective in the first place.

The authors are encouraged to design experiments that can clearly demonstrate that the proposed metrics can correctly evaluate the scenarios of (P1) and (P2), while the existing metrics, on the other hand, may fail and produce a "false" high disentanglement score. The solution of P1 is automatic, by the definition of our metrics. For P2, it would be straightforward to show that NK will pick up on $z_{\neq i}$ representing $z_i$ by XOR or something similar, as learning a function like XOR is a simple task for an MLP. We will add an experiment of this sort.

What perspective of disentanglement does the two proposed metrics are focused on? For example, using the terms "modularity", "compactness", and "explicitness" in [2] or universal? They do not fit so neatly into the framework of [2]. NK corresponds to modularity, but SNC is a mixture of explicitness and compactness.

Thank you for your time and comments. Please see our replies below.

Consider that there are two factors and z1 = c1 + c2 and z_i≠1 = 0, in this case two factors are represented by z1. The SNC will be zero. Though it is arguable about the definition of disentanglement, this toy example should be the worst case. SNC will not be zero, because a higher value of $z_1$ will make a higher value of $c_1$ more likely, and the same for $c_2$. It will be low, but that is correct this case. Whatever definition of DE one has in mind, this scenario clearly is not disentangled, as each neuron is only supposed to represent a single factor, and represent it very accurately, whereas here, $z_1$ represents two factors, both imperfectly.

The example in Figure 1 right is not clear. What is the coding map for z1? I suggest the authors to show the map between factors and the code in a table, like (square,blue)→0. Thank you. We can reformat as you suggest.

Thank you for your time and your review. Please see our replies below.

Strong DE may be too high a bar. Many existing works still target strong DE and discuss the benefits of their work wrt strong DE.

The authors dismiss the work of Locatello et. al. The main conclusion of Locatello et al. is that DE requires some training labels. We make clear our view, like that of the other recent works we cite, that, in practice, the work of Locatello is relevant but does not completely preclude work on unsupervised DE. In any case, we also include two semi-supervised models in our experiments.

In terms of the formulation of disentanglement. How are the generating factors defined? They are the labels in the datasets we use.

Are there disentanglement methods that can achieve high disentanglement levels on a broad range of disentanglement tasks with the proposed metrics? We are not aware of any tasks that are regarded as DE tasks. If you would like to clarify what tasks you are referring to, we will do our best to answer.

Thank you for your time to read and your comments. Please find our answers below.

How would your metric handle situations which are not compatible with strong-disentanglement? They are designed for strong DE only. As we argue, one cannot have a single metric that properly measures both weak and strong DE.

The Hungarian algorithm usually uses L2 over whichever values you use. You chose MI, which means you’re taking squared distances of bits in order to do tie-breaking or fine-grained factor-latent attribution. Is that really what you want? There is no L2 in the Hungarian algorithm. It simply receives as input an $M \times N$ cost matrix $C$, $M \geq N$, and returns some permutation $s \in S_N$ that minimizes $\sum_{i=1}^N C[S_N[i],i]$ (use python-style indexing). There is no mention of L2 in the original papers by Kuhn and Munkres [1,2], nor the documentation for the scipy implementation, which is what we use. The values of our cost matrix are MI, just as we describe, with no notion of squared distances of bits.

Presentation of metrics, what is $z_i$? Thanks. We will specify that we mean the latent indices are ordered so that $z_i$ corresponds to $g_i$ for each $i$.

What happens if you had continuous factors/latents? The latents are always continuous. If the factors were continuous, they could be binned in the same way that we bin the latents. Indeed, many of the labels in these datasets essentially are binned continuous values.

How does SNC differ from discrete mutual information? Both accuracy and MI are roughly used for the same purpose as performance metrics, but they are computed quite differently and are different theoretical quantities. SNC is also chance adjusted.

How do you avoid issues with overfitting by using classifiers? Note that there is a train-test split used for the MLP classifiers. We employ early-stopping and generally do not observe any overfitting. Note also that most current uses of DCI use a gradient-boosted tree.

How different is it to doing the “difference between best and next best” latents that most other metric do? Your Hinton diagrams do not have a lot of mass on more than 1-2 factors per latent (as expected if the methods work), does it really matter? Yes, it definitely matters. This is an essential part of our paper which we discussed at length. Having low MI with every neuron individually does not mean that there is low MI with the entire set of neurons. With all due respect, you criticize the paper because we spend too long describing the problems, but you seem to have misunderstood this problem despite our lengthy description.

Table 4 is more interesting than Table 1. The only difference is that Table 4 has results for a linear classifier as well. We can add this to the main paper.

Bold the best model in each row Thanks, we will do this.

Table 2 cites a CG linear which does not exist. Thanks, we will fix this.

The comparison of accuracies at the end of Section 5.1 just does not feel appropriate. The purpose is not to imply that the collective accuracy and the sum of individual are calibrated on the same scale, but rather to show how the informativeness of each individual neuron can be low, while that of the set is high. This is also the point of the XOR example. XOR itself may not be likely in practice, but the pattern of low-individual-high-collective is.

Compositional generalization results. We do not claim that these models completely fail at CG. The performance is low, which we believe is supported by our results and those of previous works, but it is not zero. There is two full pages of discussion on the CG results, and we explicitly state, at the bottom of p8 "some models achieve up to 30% even after chance adjusting, which is significantly higher than random guessing". We do not assert a clear-cut no, as you claim.

[1] Harold W. Kuhn, "The Hungarian Method for the assignment problem", Naval Research Logistics Quarterly, 2: 83–97, 1955

[2] J. Munkres, "Algorithms for the Assignment and Transportation Problems", Journal of the Society for Industrial and Applied Mathematics, 5(1):32–38, 1957