Artificial Kuramoto Oscillatory Neurons

Thank you so much for rating our manuscript with such a high score, and thank you for the suggestion regarding the runtime comparison! For details on this, please refer to the General comments. In summary, runtime overhead in AKOrN can be negligible as the model size increases. We show the runtime comparisons on different datasets in Figure 17.

on Lines 140-141, you argue that “we found that the use of C is necessary for stable trainings […] as a symmetry-breaking field”. But isn’t C also –and more simply—required for the network to receive inputs, and thus to perform any useful function?

Thank you for pointing that out, and we apologize for not providing a full explanation. We noticed that referring to training-related aspects at this point was confusing, so we removed the sentence mentioning training stability from that paragraph.

What we intended to convey is that the training became unstable when we removed $C$ and instead processed the input directly into $X^{(0)}$. This model is equivalent to just stacking Kuramoto layers and its performance is significantly worse than AKOrN (Please see Figure 16 in Section A.4 in the Appendix for details).

We are glad that you enjoyed reading the manuscript! We will answer the concerns and questions you raised one by one.

It is not clear what is the actual conceptual contribution / novelty of the Kura model proposed in this paper compared with Lowe's Rotating Feature (and recent updates). For example, equation (6) is very similiar to the "binding mechanism activation" in [1] and I have the intuition that this is the essential mechanism for the binding ability of model, similiar to [1]. If I am wrong, please correct me.

Let us first clarify that the Lowe's rotating feature (RF)’s activation is very different from just "taking norm" used in our Readout module. The activation used in RF is the sum of the two following terms: "the norm of the weighted sum of rotating neurons" and "the weighted sum of the norm of neurons". The latter term is essential in RF and called xi-term [i][ii], which enables the model to learn binding features. [ii] reported no binding is observed without xi-term (See the “-χ” of Table 3 in [ii] which shows the ablation of the xi-term). Our norm term in the Readout module is more similar to the former one: "the norm of the weighted sum of rotating neurons".

The norm term in our Readout module is used mainly for the purpose of stabilization of training. When AKOrN is trained on CIFAR10 without the norm term, it significantly underfits the data, resulting in a 7% drop in accuracy compared to the full AKOrN model (See Figure16 and Section A.4).

As an ablation study, we test a model replacing the Kuramoto mechanism with the conventional residual update. We see the model degrades significantly in both the object discovery performance and Sudoku solving, as shown in the table below, which clearly shows the large Kuramoto model’s contribution to the performances. The table is included in the updated manuscript as Table 6.

CLEVR-Tex

Sudoku

At a high level, Rotating Features do a simple forward pass through the model, where the orientations of the features are set statically via the rotation bias, and feature binding is through a specialized activation mechanism called xi-binding. In the Kuramoto model, we explicitly model the temporal component by implementing an iterative procedure within each layer that promotes synchronization.

---

When generalizing the original Kura model to high-dim, it is not clear whether the Proj in eq(2) is an 'equivalent' generalization of sin(θj−θi).

We show the relation between the vectorized Kuramoto model and the original one in Appendix A.1 along with a visualization of the projection operator. In summary, the vectorized Kuramoto model includes the original one in a special case where $N=2$, $c_i=0$, and the connectivity $J_{ij}$ is a scalar multiple of the identity matrix.

sin(θj−θi) is a essential mechanism in original Kura model to guarentee the synchrony as a stable state and sin acts among each pair of neurons before the ∑, but Proj only acts after the ∑.

The order of the application of ∑ and Proj does not matter because Proj is a linear operator (${\rm Proj}_x=I - xx^{\rm T}$) and thus Proj ∑ = ∑ Proj. The same is applied to the original Kuramoto model: $\sum_j {\rm sin}(\theta_j−\theta_i) = Im [\sum_j e^{\sqrt{-1}(\theta_j−\theta_i)}] = Im [e^{-\sqrt{-1}\theta_i} \sum_j e^{\sqrt{-1}\theta_j} ]$

What is the contribution of the "Proj", and will there be synchronization, e.g. given input image, if we replace Proj into general linear sum (ablation of Proj)?.

Thank you for your suggestion! We use Proj to align the vectorized version with the original Kuramoto model, as we describe in the above comment. Proj ensures the update direction stays on the tangent space of the sphere. Note that with or without Proj only changes the length of the actual update direction of each neuron. The updated $x_i$ stays on the sphere since we normalize the updated neuron to be the unit vector in Eq (5).

We test AKOrN without Proj operators and summarize the results below. We see almost identical, and slightly degraded performance on the CLEVR-Tex object discovery experiment and on the Sudoku solving, respectively. Interestingly, without projection, the AKOrN’s adversarial robustness and uncertainty quantification get worse compared to the original AKOrN.

(a) CLEVR-Tex object discovery

(b) Sudoku board accuracy

(c) CIFAR10 classification

Since the activity is constraints on the sphere, it loses the representational ability for encoding feature presence

Thank you for your comment! We would like to note that although each single oscillator cannot encode the feature presence, a group of oscillators can learn such feature presence through the degree of their coherence: e.g. we can think of the magnitude of the sum of a set of oscillators as feature presence. This way to encode feature presence is possible through our readout module that takes the norm of the weighted sum of oscillators.

On the experimental side, it mostly ignored the Lowe's model for comparison (Fig.3, Fig.4,Fig.5,Tab.1,Tab2, Tab3...) and other synchrony based model

The Lowe's Rotating features [iii] and other synchrony-based models’ [ii, iv, v] results are shown in Tab. 14 in the appendix, and AKOrNs are consistently better than these models on all synthetic datasets, except for dSprites on which AKOrN and Rotating features are on par. We show the Rotating features (RF)’s results on PascalVOC on Table 18, and AKOrN is significantly better than RF. Note that no synchrony-based models besides AKOrNs are directly applicable to natural images. Even on CLEVR data, which is one of the synthetic datasets, Lowe’s model and other synchrony-based models get considerably worse performance than slot attention and AKOrN (See also Figure16 in Lowe's work [iii] where the model separates the objects based on low-level features such as colors). The model in [2] the reviewer suggested is only tested on simplistic, synthetic data, and it would be unlikely to scale to complex datasets such as CLEVR, CLEVR-Tex, and natural images. However, if you would still like to see the results of [2] on these datasets, we are willing to train [2]’s model on our dataset and report the performance.

The code is not provided, so it is hard to evaluate the the reproducibility of the results.

We are currently preparing the reproducing code and will provide a part of the code for the object discovery experiments as soon as possible. We are committed to open-sourcing the entire codebase, including scripts for reproducing all the experiments conducted in our work, at a later date.

The author misses several essential recent works: e.g. [2] shows how oscillation emerges in spike-based model to bind features; [3] combine Kura model with ANNs and also achieved synchrony binding.

Thank you for the references! We apologize for having missed these works. We discuss them in the related work section of the updated manuscript.

The first two sentences in Abstract is quite confusing and the first paragraph in Introduction is not very informative.

Thank you for your comments! Could you elaborate a bit more about why the abstract is confusing, and the first paragraph is not informative? We are a bit confused about this comment. The first two sentences in the abstract describe the core idea and motivation behind our proposed neuronal mechanism. The first paragraph of the introduction mentions the McCulloch-Pitts neuron mechanism which is the current main paradigm of deep neural networks but ignores the temporal dynamics of the real neurons. The paragraph explains the very motivation of our temporal processing neurons that is inspired by the recent findings in neuroscience.

• Regarding references to the claim that we made in the first sentence of the abstract and the first paragraph of the motivation section

(This is our response to the reviewer's comment "Can author clarify the reference if such claim is well-documented?")

Thank you for your suggestions! There are many, but let us name a few. We include these references in the motivation section in the updated manuscript.

• Amari S. (1982). Competitive and cooperative aspects in dynamics of neural excitation and self-organization. In Competition and Cooperation in Neural Nets: Proceedings of the US-Japan Joint Seminar held at Kyoto, Japan February 15–19, 1982 (pp. 1-28). Berlin, Heidelberg: Springer Berlin Heidelberg.
• Grossberg S., and Grunewald A. (1997). Cortical synchronization and perceptual framing. Journal of cognitive neuroscience, 9(1), 117-132.
• Gray et al. (1989). Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature, 338(6213), 334-337.

We also experimentally show results aligned with the claim: the deeper the layers, the more object-aligned binding features the model learns (See Tables 15,16 and Figure 29).

Can author shows the segmentation results on binary datasets, like Zheng2024. The different color of object may make it eaiser to break symmetry compared with binary images…

(Nov. 21, 11:34 GMT: We realized that we had forgotten to include the performance table and reference the figure showing the clustering results in this rebuttal comment. We have now added them. We apologize for overlooking these details.)

Thank you for the suggestion! We manually created a binary image dataset, named “Shapes,” consisting of images with 2–4 objects that are randomly sampled from four basic shapes (triangle, square, circle, and diamond). Note that each image can have multiple objects of the same shape together. The table below shows that AKOrN is considerably better than its counterpart model (ItrSA). The same table is included in the updated manuscript as Table 15.

(a) FG-ARI

(b) MBO

Figure 24 demonstrates that the AKOrN model learns significantly better object-binding features than ItrSA on this highly symmetric dataset. Furthermore, having more layers benefits the separability in AKOrN while doing little for ItrSA.
The primary reason we do not achieve perfect results here would be the use of patchified images, which makes it difficult for the model to assign fine object details to coarse features accurately.

Fig.4 the segmentation is more likely based on color instead of object-instance (the highlight is treated as an object).

We would like to emphasize that while the model is actually distracted by the strong highlight, the objects in Figure 4 of the initial submission are very similar in color, which highlights the opposite of the concern: the segmentation is unlikely to be based on the objects’ color. The results on highly symmetric datasets, as mentioned above, further support this point. Additionally, it has become evident that AKOrN has learned improved object-binding features thanks to the newly introduced up-tiling method. In the updated manuscript, Figure 4 on the right side shows the AKOrN is not misled by the shiny, object-like white highlight on the floor, even in a scene where both the objects and the floor are white.

We observe that deeper and wider AKOrN models are less misled by such superficial features. Figure 29(b) shows that a larger AKOrN model is not deceived by an irregular floor pattern, a mirroring object, and a highlighted object (1st, 2nd, and 5th row, respectively).

Besides, what is the actual input dimension of the image to the model? To what extent it is downscaled?

Here is a table that summarizes the size of the images, the number of channels, and patch-size (= the downscale factor). The same information is shown in Tabs 9-11 in the Appendix.

Dear Reviewer AtLx,

We here provide the code to reproduce the CLEVR-Tex results:

https://anonymous.4open.science/r/akorn-049D

We hope this will help clarify and validate the results presented in our work. Please feel free to reach out if you have any further questions or require additional details.

Best, Authors

Thank you so much for your valuable comments and suggestions, which are greatly helpful in improving our paper quality. We address your questions and concerns below, one by one.

Is the idea that AKOrN is a highly competitive model for models 'trained from scratch', and that comparing it to models with some pretrained parameters (e.g. Lowe et al. 2024) is unfair?

Indeed, we believe the direct comparison to models that are applied to large-scale, pre-trained models is not fair. AKOrN achieves state-of-the-art object discovery results on real-world images while being trained from scratch. In contrast, to the best of our knowledge, none of the existing object discovery methods perform considerably better than random in this regime. Note that in Table 1 and Table 18 in the Appendix section, we show that AKOrN now is better than such models with pre-trained parameters including (Lowe et al. 2024), DINASOUR, Slot-diffusion, and SPOT on Pascal, and better on COCO than those methods except for SPOT.

there is a notion from the abstract and sections 1/2 that this work is relevant to neuroscience, but the link feels quite tenuous to me. Is it reasonable to think of neurons as high-d points on a sphere?

line 36: 'we follow a more modern dynamical view of neurons as oscillatory units'. Is this correct? ... I would say that the majority of computational neuroscientists currently model neurons as scalar values.

People modeling oscillatory neurons often represent them by complex values (or equivalently by the sinusoidal function), especially when they investigate synchronization in neuron populations (e.g. [i]). The complex-valued neuron is quite close to our generalized Kuramoto oscillator with N=2. The N=2 model is equivalent under certain symmetric constraints, as we commented in the responses to reviewer AtLx. The AKOrN model with N=2 exhibits good adversarial robustness. When we extend the Kuramoto oscillators to N>2, we see an improvement in the model capacity and flexibility and achieve good performance on object discovery and Sudoku solving. In the end, we take inspiration from neuroscience and physics, but we take the liberty to deviate from these inspirations whenever this can lead to improved results. For example, we also do not implement spiking neurons, as this would be much harder to train and scale. We sincerely hope that by showing that the dynamical view of neurons improves performance in a wide spectrum of tasks, our paper will motivate following subsequent work not only in deep learning and AI but also in neuroscience and physics.

For table 3 results are presented with error bars over 5 seeds (which I favour), why is this not the case for any of the other results presented?

We will try to repeat experiments with different random seeds. However, because of our limited computational resources compared to the number of tasks and models we tested in our work, it will take some time to complete and that is why we could compute mean and stds for the Sudoku task, where the training completes much faster than the other tasks. We will report the error bars of each performance as much as possible until the end of the rebuttal phase.

could you clarify to me the exact novelty of the model? Is it that it's not been introduced in deep neural nets before?

Thank you for the comment! As far as we know, the original Kuramoto model and its multidimensional model have not been used in deep nets before (this work [ii] uses the original Kuramoto model for clustering but uses a single layer (the connectivity is predicted by a conventional multilayer neural network) in very synthetic cases). The model’s novelty is as follows:

• The use of the multidimensional Kuramoto model. (Table 7 and Figure 14 in the updated manuscript show that models with N=2, which is close to the original Kuramoto model, significantly underfit in the object discovery experiments and the Sudoku solving.)
• The building block we propose. It is composed of a pair of the K-layer and the readout module, which together process the conditional stimuli and oscillators. Without norm terms, for example, the model is unstable and again underfits the CIFAR10 classification problem (Figure 16). The same figure shows that just stacking Kuramoto layers without conditional stimuli further underfits the task.
• The overall network architecture with the proposed blocks. We solve a wide variety of tasks with a shared architecture and show improvement over transformers and conventional ConvNets.

Our work’s novelty is not limited to the model architecture. We demonstrate that the binding and neural synchrony-based idea, which have been shown effective only on relatively synthetic tasks, can be extended to natural images in object discovery tasks with competitive performance to the well-used slot-attention, and that also it proves to be effective in very different tasks such as reasoning and robustness.

line 308 'our..models differ from these approaches in various aspects such as...' should really provide all fundemental differences beyond some examples. In fact, in any case, what's the difference between 'asymmetric connections' and 'their symmetry breaking term'?

Asymmetric connection means J is an asymmetric matrix. Usually, as in the standard Ising model and Kuramoto model, $J_{ij}$ is symmetric: $J_{ij} = J_{ji}^{\rm T}$. This symmetry is important to guarantee the model’s update to minimize a certain energy function. For example, Energy transformer[iii] introduces a symmetrized attention mechanism to have such a guarantee. However, we find such symmetric models basically worsen the performance compared to asymmetric ones (See the response to reviewer AtLx https://openreview.net/forum?id=nwDRD4AMoN&noteId=dPzqDmDXMp). The symmetry-breaking term is $C$, which is a data-dependent term and not introduced in the "Hopfield networks is all you need"[iv] and [iii] and found to be important to stably train AKOrN on the CIFAR10 classification task (Figure 16). Thank you for the suggestion! We extend the discussion of [iii] and [iv] in the updated manuscript.

line 123 (and other places): what's C? the size of the neural population? this should be clarified

Sorry for the confusion. Yes, it’s the number of oscillators. We added the description of what $C$ is in the latest manuscript.

line 245: what is 'proper' energy?

The energy is proper if the differential equation is guaranteed to minimize that energy. In our case, where we allow asymmetric connection in $J$, Eq(2) is not guaranteed to minimize the energy defined in Eq(3). However, it is empirically shown that Eq(2) moves the oscillators roughly in the decreasing direction of Eq(3) (Please see animations in Supplementary Material, which includes plots of energy values over timesteps demonstrating this behavior).

line 323: I don't understand how these convolution/self-attention layers are iterated upon. What are the shared parameters?

The parameters are shared across Kuramoto steps within the same layer but differ across different layers. For example, there are three layers in the network trained on ImageNet used in the object discovery experiment, and each layer has its own learnable parameters for both the Kuramoto layers and the Readout layers. We noticed that we had forgotten to put the layer index on $J$ and $\Omega$ in Eq(4). We fixed it in the updated manuscript. Thank you for raising the question!

I didn't understand the implementation for the sudoku task. 1. is the entire 9x9 soduku grid fed all at once, or one digit at a time?

Sorry for the confusion. The former is what we did. We see each Sudoku grid as a 9x9 resolution image and process it similarly to the AKOrNs used in the object discovery experiments.

how come x_i takes the value c_i when a digit is given, does the model no longer follow equation 2? Or is this just the initial value taken before the Kuramoto steps?

We apologize for the lack of explanation about this. $x_i$ is initialized as “a normalized $c_i$”: $c_i/||c_i||_2$ if a digit is given, since $c_i$ itself is not constrained on the sphere. Otherwise, $x_i$ is initialized randomly from the uniform distribution on the sphere. This initialization is done only once to create the initial $X$ fed into the network, and every $x_i$ is updated following Eq. (4) and (5) after the initialization. We fixed the sentence describing this initialization in the updated manuscript.

line 425: 'the energy is a good indication of the solution's correctness'. Why? I can believe that stability is often a desirable thing in tasks but think this line deserves some elaboration

Please kindly refer to the General comments, where we elaborate on how the energy can indicate the correctness of the solution.

would it be possible to give a brief description/overview of what 'common corruptions' are?

Thank you for your suggestions! We include an example image of each corruption type in Figure 21 in the Appendix. The images are artificially generated corruptions that mimic naturally occurring perturbations, such as blur, weather, digital noises, and so on. Please refer to this work [v] for details.

why is it surprising that confidence and accuracy are linearly correlated?

Thank you for the comment! We rephrased it with almost perfect correlation. Usually, neural network models without regularization are overconfident in their predictions, as we show the ResNet's confidence vs. acc plot at the left bottom in Figure 9. On the other hand, regularized models can be the opposite and be too conservative, like the models on the top rows in the same figure. Our model achieves very high uncertainty quantification without any explicit regularization, which is rarely seen.

Thank you for your comments and feedback, and for substantially raising the score! We are delighted that our response has clarified your questions.

Regarding the error bars; we conducted object discovery experiments with different random seeds for the weight initialization and show the mean and std of the object discovery performance metrics in Figure 3 and Table 1 in the main sections as well as Tables 14-17 in the appendix in the latest manuscript. We see that any updated values do not affect our claims.

Because we have less than one day before the revision deadline, we will not be able to add error bars for the object discovery experiments on natural images or the image classification task on CIFAR-10. However, for those tasks, we believe that models initialized with different random seeds will not exhibit significant performance differences that could impact our claims. This is because the model for the natural image object discovery is trained on ImageNet, which contains a relatively larger number of images compared to other datasets, leading to smaller variance across differently initialized models. Additionally, the distinction between AKOrNs and conventional models in the image classification task is significantly evident in performance metrics such as adversarial robustness and uncertainty calibration.

We thank you again for your suggestions and feedback, which greatly improve the quality and clarity of our manuscript!

Thank you very much for the review and suggestions! We address your concerns one by one.

• Judging from the method, AKOrN seems to be much more computationally costly than the usual thresholding units. ... showing the training/inference time would be helpful.

Thank you for your suggestion, and please see the General comments regarding this matter. In summary, the runtime overhead in AKOrN can be negligible as the model size increases. The runtime comparison including both training and inference is shown in Figure 17.

• the paper shows that the AKOrN models show better robustness, but it is not clear from the design that they are more robust. Thus analysis of the network dynamics on some toy tasks might be helpful.

We conducted an analysis of the oscillator states of networks trained on CIFAR10 and found that oscillators highly fluctuate over timesteps, which we could think prevents the model from learning noise-sensitive features. Please see the General comments for details.

Finally, seemingly these three questions and concerns are closely related together, so we answer in the following altogether.

• the model might not work as well on more "classical tasks" such as image classification.
• why the model shows superior performance in the tested task is not well understood.
• what tasks are probably more suitable for this method and why?

While several aspects of the models are not fully uncovered (such as adversarial robustness), the efficacy of the object discovery and combinatorial problems is intuitive. The Kuramoto model promotes synchronization in oscillatory neurons, where synchronized neurons form a cluster. These clusters are then processed by subsequent Kuramoto updates that further generate more abstract features. This idea is a motivation to use the model to learn binding object features, as each object can be seen as elements that are clustered, mimicking the visual recognition process where objects are grouped based on shared features. The use of the Kuramoto model for reasoning tasks is also reasonable as the Kuramoto model can be seen as a continuous Ising model. The Ising model has been used to solve combinatorial problems. To address your questions regarding suitable tasks, we believe our model is well-suited for solving complex visual reasoning tasks. This is because the ability to learn abstract (object) features and reason over those features is crucial for such tasks. Studying our model's efficacy in such tasks is left for future work.

• Why the energy value indicates the solution's correctness? (by reviewers AtLx and ynsZ)

(Note: $x_i, y_i, c_i$ and $J_{ij}$ are multidimensional vectors and a connectivity matrix, respectively, even if they are written in non-bold style. It seems that ICLR's markdown cannot process bold symbols for some reason)

Although the update is not guaranteed to minimize the energy, we still have a good interpretation of what the value in Eq(3) means. Please see the Figure 11. in the updated manuscript. The figure shows that the negative energy $\langle c_i + \sum_j J_{ij} x_j, x_i \rangle$ and the length of the projected update are inversely proportional given the length of $y_i:=c_i + \sum_{j} J_{ij} x_j$. More specifically, the eq(2) can be rewritten as

$E(X) = - \sum_i c_i^{\rm T} x_i - \sum_{ij} x_i^{\rm T} J_{ij} x_j = - \sum_i ( c_i + \sum_j J_{ij} x_j)^{\rm T} x_i = -\sum_i \langle y_i, x_i \rangle$

Thus, the negative energy tells us how the update direction is aligned with the current oscillators' state. If it’s small, the length of the projected update in the oscillators is expected to be high, and vice versa. From this interpretation, we see the energy as a sensible indicator of how the oscillators settle down, especially for the Sudoku problem where the solution is well-defined. It is natural to consider that low-energy states represent the model’s confidence in the solutions, as these oscillator states are less likely to undergo further updates. However, if the model fails to learn a good energy landscape, then even low-energy states may correspond to incorrect solutions. This incorrectly learned energy happens when we do not use the natural frequency term $\Omega$ (See Figure 26 and Table 18 in the Appendix). The natural frequency term, which is a completely asymmetric transformation, serves as a form of exploration in the energy landscape, which would allow the states to escape from bad minima.