• Missing comparisons to related work that uses delay differential equations for designing RNNs; e.g., the $\tau$-GRU proposed in https://arxiv.org/abs/2212.00228, and also other recent models (e.g., LEM, coRNN, state-space models)
• Missing evaluation on important benchmark tasks such as sequential image classification on MNIST and CIFAR-10
• Missing ablation studies on the delay factor $\tau$ (what is the role of it?)
• Missing details on the activation functions written in Eq. (1) (what are $\sigma$ and $\phi$?)
• Missing theoretical results to show that the proposed model indeed has better representation learning capability

Cons:

• The experiments are primarily based on outdated RNN architectures and miss breakthrough time-continuous architectures of the past few years (Gu et al. 2021, Rusch et al. 2021, and Rusch et al. 2022).

References

• Gu et al. 2021, Efficiently Modeling Long Sequences with Structured State Spaces.
• Rusch et al. 2021, Unicornn: A recurrent model for learning very long time dependencies.
• Rusch et al. 2022, Long expressive memory for sequence modeling.

This is not the first continuous model in the sense of evolving the hidden state without jumps, neural CDEs are one example. It seems that the difference to neural CDEs is the network that you use, in particular you have GRU updates, and you add a single previous step from the solver.

This is also not the first neural DDE. The paper mentions two previous works but does not explain how is the proposed method different, nor is there a comparison with these methods (or explanation why it's not possible).

Although the method is presented as continuous, the main experiments are encoding fixed history and making a prediction based on the encoded vector. So the method is still encoder-decoder, therefore, you should also compare to mTAN [1] or potentially some other strong baselines. Figure 1 (c) shows that the values x(t) are continuous in time but this is only available up until the last observed data point, unless you use causal interpolation as discussed in [2]. Note that the causal cubic Hermite spline will still not allow online prediction. The method is therefore still lacking the online prediction that is possible with "jump" methods.

The implementation in code uses saving and loading previous delay values from a file which is not ideal, I can imagine some errors arrising because of that. Also, the authors should try to have a better implementation if they want wider adoption of their method.

The paper could be written better in general.

Minor:

• Limitations 1. and 2. in Introduction are not really limitations but simply design choices.

• "Eq. equation" in many places

• Algorithm 1 is not very useful, it contains too little information about the actual training procedure.

[1] Shukla & Marlin, Multi-Time Attention Networks for Irregularly Sampled Time Series (2020)

[2] Kidger et al., Neural Controlled Differential Equations for Online Prediction Tasks (2021)

1. The authors overlook several closely-related works, warranting a more comprehensive discussion of the relevant literature.

2. The introduction section lacks a clear and well-articulated explanation of the underlying intuition behind the proposed model.

3. The experimental results may benefit from greater credibility by including additional baselines that are currently absent from the analysis.