Learning diverse causally emergent representations from time series data

Many thanks for the feedback. We apologise for the lack of clarity in some of our explanations. We will improve these and add pseudo-code in the camera-ready version of the paper.

Regarding the concern about not having enough examples, as we describe in more detail below and in the global rebuttal, we have conducted several additional benchmarks and evaluations. These new analyses include:

1. Two new real-world datasets, showing our method works well in both cases.
2. One more synthetic dataset, Conway’s Game of Life, which is arguably one of the most well-known examples of emergent behaviour. Since in this system there is a notion of what the emergent feature should be (known in the literature as “particles”), we can show that the learned feature captures the state of these particles.
3. Comparisons against baseline algorithms, making the case that our method finds features that standard algorithms based only on temporal predictions do not.

In response to your questions:

How does Figure 4 prove that an emergent feature was found? What is the quantity on the x-axis in this figure?

We apologise for the oversight in the labelling of this figure. The X-axis represents training time, and the Y-axes represent the emergence measure $\Psi$ and its adjusted variant $\Psi_A$. The plot proves that an emergent feature was found because the emergence measure is greater than zero, which is a sufficient condition for the feature being emergent (as shown by Rosas et al., 2020; Ref. [30] in the paper).

Can you share the code of the architecture and its training or describe it in more detail?

Due to the constraints of the double-blind review we have been unable to share a Github link, but we will include it in the camera-ready version of the paper. Also, we apologise for the lack of clarity in our description – we will improve the description of the method and add a pseudo-code algorithm to the paper in the camera ready version.

Would you expect this method to find emergent features in standard, well-understood systems, such as Ising models?

Although the behaviour of the Ising model is often described as emergent, our method is not readily applicable to the Ising model. This is because our method is explicitly designed for time series data (as the title indicates), but the Ising model doesn’t have time per se – the Ising model specifies a probability distribution over an ensemble of spin configurations, without any dynamics. There are of course several options to add dynamics to the spins (e.g. Metropolis-Hastings, Glauber, Kawasaki, etc.), but none of these are an integral part of the canonical Ising model. For this reason, if one were to choose one of these dynamics and apply our method, then the kind of conclusions one may draw is whether or not e.g. Glauber dynamics display emergent features, and not about whether the Ising model does.

For this reason, we have chosen instead a different well-understood system that has dynamics and is widely considered to display emergence: Conway's Game of Life. As mentioned above, we show that our method can successfully find emergent features from Game of Life time series and that these features capture the state of the particles in the system.

Can you explicitly write down the prediction-only objective in Fig.2?

The prediction-only objective is $I^\text{S}_\varphi(f\_{\theta}(X_t); f\_{\theta}(X\_{t+1}))$, i.e. the mutual information between the past and future of the learned representation. This quantity depends on the parameters of the critic $\varphi$ and the parameters of the representation learner $\theta$, and is calculated using the SMILE estimator as defined in Eq. (4).

Finally, in response to the limitation:

There is no open access to the code.

As mentioned above, we didn’t include a link due to the constraints of the double-blind review. We will add a link to a publicly accessible Github repository with all the code once the requirements of double-blind review are lifted. In the meantime, we have sent the AC an anonymised link to the code for the model architecture.

As mentioned in the overall rebuttal, we have now conducted a wider range of evaluations, both using the same architecture on more datasets and using other architectures on the same datasets.

• Specifically, we add two more brain activity datasets which capture different aspects of neural dynamics: fMRI, with high spatial resolution and low temporal resolution; and MEG, with low spatial resolution but high temporal resolution. Our method is able to find emergent features in both cases.

• We ran experiments on one more synthetic dataset used in Rosas et al. 2020, Conway's Game of Life. Here we found that our method can learn emergent features, and that these can be interpreted as encoding the state of the particles, as one would expect.

• We compare against standard RNN and MLP architectures that predict the future state of the time series, without any information-theoretic loss function. We confirmed that neither the RNN or the MLP are able to learn emergent features.

To the best of our knowledge, there are no established baselines for discovering causally emergent representations – if anything, the Game of Life is one of the most long-standing canonical examples of emergent behaviour generated by relatively simple rules, and we now show our method works on it. We would of course be happy to consider any other suggestions.

Finally, in response to the limitation:

The authors did not explicitly state the limitation of the paper.

We apologise for the lack of clarity on this regard. We state some of the limitations of our work in Section 4, although we acknowledge this needs to be more clearly laid out. For the camera-ready version we will add an explicit discussion of our method's limitations.

As mentioned in the global rebuttal, we have now conducted extensive additional evaluations, including:

• Two new real-world datasets and one new synthetic dataset;
• New analyses on the existing synthetic dataset with lower correlation coefficients; and
• Comparisons against baseline algorithms based only on standard temporal prediction.

Overall, our method was able to learn emergent features in all these new scenarios, the resulting features were interpretable for the synthetic cases where a notion of “ground truth” emergence is known, and baseline algorithms (standard RNN and MLP) do not learn any emergent features.

In response to your questions, we have fixed all the typos and rephrased the relevant sentences in the paper. Here are responses to the rest of the questions:

Emergence comes in many forms – I wonder if you could discuss alternative definitions / alternative perspectives on the matter?

We totally agree that emergence comes in many forms, and we acknowledge that our paper could do a better job at placing our definition within the broader literature on emergence. For the camera-ready version we will include a discussion regarding alternative definitions of emergence, and how this could be explored in future work.

Eq (2): can you motivated the adjustment term further the summand $(n − 1) \min_i ...$ ? Are there alternatives to this that would be more targeted towards actually identifying true redundant mutual information?

The reviewer is right in that the discussion surrounding the adjustment term could be expanded. For the camera-ready version we will include a proof of where this adjustment term comes from, and a discussion including what redundant information is and what role it plays in the formula. As a matter of fact, there are multiple measures of redundant mutual information in the literature, and a comparison in this context would be an interesting avenue for future work. However, we would like to emphasise that the current form of the adjustment is a recognised and well-defined measure of redundant information (in the sense that it satisfies many of the natural properties one would require of such a measure), as shown in the following paper:

Barrett, Adam (2015). Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems. Phys. Rev. E 91, 052802. doi: 10.1103/PhysRevE.91.052802

Line 190ff. removing the macroscopic MI term seems not to save much – or does it? The observation is interesting, but I wonder if the authors want to make a computational argument here as well.

We agree, thanks for this observation. To clarify, there are in fact two arguments for doing this: one is the computational argument to save some compute; and the other is that in some cases we observed it stabilised learning dynamics (our hypothesis is that it alleviates the adversarial relationship between the different components of our objective). For the camera-ready version we will clarify these issues and include both arguments.

About the biological dataset – this is very ad-hoc somewhat. What does this analysis tell us really except that there are some complex spatio-temporal statistical correlations in the data? I find this one marginally useful.

Our motivation to study emergence (as described in the introduction) is to understand how cognitive systems encode information in macroscopic variables, which naturally motivates us to test our methods on data from biological brains. The fact that the same method that successfully identifies particles in the Game of Life detects emergent dynamics in brain data is extremely encouraging, suggesting that neural systems may encode information into “glider-like” collective features. This paper provides direct empirical evidence supporting this idea. Moreover, it is particularly encouraging that the emergent character is observed in multiple neuroimaging modalities that cover multiple spatial and temporal scales of the brain's dynamics.

Needless to say, there is much future work to be done examining precisely what the role of emergent dynamics in the brain is, and how these emergent features contribute to brain function. We see this as a longer-term research programme, which our method is enabling for the first time. Since we do not have a sense of “ground truth” of what the emergent features of brain activity should be, this is difficult to do as part of this paper (although note that for cases where such ground truth is known, such as the bitstrings and the Game of Life, the results align with our intuitions). For the camera-ready version, we will emphasise the significance of these findings and the need for future work interpreting the learned emergent features in real-world data.

Finally, in response to the limitation:

Unclear how robust this is as ablations and comparisons are not very extensive.

As mentioned in the global rebuttal, we have now performed:

1. Further ablation analyses, showing the expected behaviour that key properties of the architecture (e.g. ability to learn diverse features) are lost if the corresponding elements of the objective are removed; and
2. Further comparisons with other, non-information-based architectures (e.g. RNNs, MLPs).

In response to the specific weaknesses identified:

It’s not immediately clear how to interpret results. The paper shows figures, but it doesn’t explain them much. Interpreting them requires a lot of re-reading the methods section

We apologise for the lack of clarity in our explanations. For the camera-ready version we will make sure the figures are explained in more detail and the text flows more smoothly.

On the ECoG dataset, giving intuition/semantic understanding of emergent features (or at least attempting interpretation) would be cool

We enthusiastically agree with the reviewer that obtaining a semantic understanding of the learned emergent features in ECoG would be definitely cool. However, we see this as a longer-term research programme that our method is enabling. Since we do not have a sense of “ground truth” in terms of what the emergent features of brain activity are, this is difficult to do as part of this paper.

In cases where we do have a reasonable sense of what these features might be (e.g. in the synthetic dataset and in Conway’s Game of Life), we conducted post-hoc analyses to interpret these features, and found that they align with our intuitions. For the camera-ready version we will emphasise the need for future work interpreting the learned emergent features in real-world data.

Limited experiments on real data in general - ultimately, only one experimental setting is shown as far as I understand. The synthetic problem, while useful, is simple

As mentioned in the global rebuttal, we have now added experiments on two more real-world datasets, and demonstrate our method can successfully learn emergent features on both. We also add one more synthetic,but far less simple, dataset using Conway’s Game of Life system, and use our method to learn emergent features from it and interpret them with post-hoc analyses.

Lacking baselines or extensive comparison to existing methods, even if purely information-theoretic

In addition to new ablation studies, showing comparisons against other information-theoretic loss functions, we now compare our method against a standard RNN and a standard MLP architecture (without any information theory). The results suggest that RNNs and MLPs do not learn emergent features on their own, supporting the value of our method.

It would help to have clearer comparison to existing methods so that we could see the value-add of this innovation, not just the novelty and value alone

We agree that comparisons against existing methods can make a clearer argument in favour of our paper. As mentioned above, we now show that standard RNN or MLP do not spontaneously learn emergent features. Additionally, we show in post-hoc analyses that the combination of the learned emergent feature of our method and an RNN representation enables better predictions than either of them in isolation (informally: a small RNN plus an emergent feature is better than a large RNN), making a direct case for the value-add of our method.