On the Dynamics of Learning Time-Aware Behavior with RNNs

• The scenario tested is very simple and hence generalization of the findings is questionable.

• While the author elucidates the statistical aspects comprehensively, a more detailed explanation of the autoencoder's use would have been beneficial. For instance, discussing the motivations behind its selection, why it's deemed the best choice, or testing its performance against alternatives like variational autoencoders.

• In the experiment section, the author evaluates the results using five datasets from the Monash Time Series Forecasting Repository based on dimensions, frequencies, and length. The chosen samples appear to be of a small size. Merely calculating the average rank might not provide an unbiased and comprehensive evaluation. The results would be more persuasive if the author utilized a broader range of datasets and presented a critical difference plot.

• The author might consider employing other techniques, such as artificially creating uneven sampling frequencies, to garner more samples.

• My main concern about the work is that the tasks considered in this paper are rather artificial or synthetic. It is unclear how the results transfer to real-world datasets, which are more nuanced and may contain a superposition of multiple overlapping periods.
• Another weak point of the work is how the observed results translate into building better RNN models and learning algorithms. Specifically, the paper does a great job characterizing the dynamics but how can this information be used to improve our methods
• Finally, I think the paper would be stronger if it also considered more modern RNN architectures. Particularly, there have been some breakthroughs in the past few years (see Rusch et al. 2021, 2022, and Gu et al. 2021).

References.
Gu et al. 2021, Efficiently Modeling Long Sequences with Structured State Spaces.
Rusch et al. 2021, Unicornn: A recurrent model for learning very long time dependencies.
Rusch et al. 2022, Long expressive memory for sequence modeling.

• The paper is extremely difficult to read and understand because most of the sections do not include a broad high level context and directly dive into technical details with math/notations. In order to make the results widely accessible to the machine learning community of ICLR, most sections of the paper need to be re-written with appropriate context.

• The related works is minimal /missing. Several prior works that have attempted to understand neural network dynamics, and in particular those that have tried to understand the dynamics of recurrent models need to cited, and discussed. It will help to have a separate related works sections for this.

• The experiments are on very toy settings. While they are helpful in understanding each hypothesis, I am not sure if any of the claims will hold up in tasks with real datasets. It may be helpful to have at least one such experiment where the data is not generated in the paper, and is based on an existing sequence-to-sequence modeling task.

• The task and settings are designed as a simplified framework for studying the influence of time, and some may argue that they are overly simplistic. While this setup allows for a comprehensive exploration of the system's behavior, I remain unconvinced that learning a two-state automaton (TA) truly represents a meaningful real-world sequence learning challenge. I am uncertain about the extent to which the observed results can be applied to real-world datasets.

• The paper concentrates on a particular RNN architecture and a specific activation function, potentially constraining the applicability of its findings to other architectures and tasks. Since learning dynamics are closely tied to the chosen RNN architecture, I believe the paper lacks comprehensiveness without addressing potential variations across RNN types. Numerous RNN architectures have been introduced that handle temporal information differently from a standard RNN (or GRU/LSTM).

• In Figure 4, it's unclear which parameters and their corresponding values led to the observed bifurcation. Given that the time-dependent system in this paper relies on inputs, these inputs can also influence the Jacobian matrix and subsequently impact the stability of the fixed points over time. Therefore, additional information is required to better understand the discussed bifurcation.

• Regarding the sentence (page 18) "These trajectories are quasi-periodic but approach period P as training progresses", it's not entirely clear how we can verify that the observed behaviors are genuinely quasi-periodic. They could also be higher-order periodic orbits converging towards a lower period P as training advances. One method to confirm this statement is to delve into the Lyapunov exponent spectra and the maximum Lyapunov exponent. Otherwise, how one can validate this assertion?