Recurrent Complex-Weighted Autoencoders for Unsupervised Object Discovery

The colors in the polar plot visualization are supposed to reflect the phase value matched with the colors in the phase maps. However, why do some phases of similar angles have different colors? Is it possible to use a standard colormap applied to each plot to visualize the phase distribution and compare between images?

Please note, that the colors in the phase maps are matched (between radial plot and heatmap) only within a single row and not across rows. Regarding the standardization of colormap across different image samples (rows), we have not managed to achieve that. We understand that having uniform colors across all rows would be nicer. We would be happy if the reviewer can give us some pointers for the same? Thanks for your feedback on this!

The t-SNE to visualize the phase map first ensures that the magnitude of the activity is set to 1 (L562). However, the result from the clustering exhibits an activity with magnitude differing in their amplitude value. Why is this the case?

Thank you for bringing up this point about the magnitudes in the radial plot visualizations. Please note that our phase map visualization is an extension of similar plots in prior works such as CAE[23] (see Figure 6 in [23]). In their plots we can see in the radial plots the true magnitudes are shown along with colors that match the orientations (phases) from the heatmap. We have tried to maintain stylistic consistency in the radial plots in [23] and ours.

limitations focus only on the reliance of SynCx on color cues, even though this model performs better than the baseline in this experiment.

This experiment highlights a flaw in the general class of synchrony-based models. Please note that our model performance also drops on the grayscale CLEVR compared to the original colored version (shown in Table 1). We believe this is a valuable insight that poses an open problem in this class of synchrony-based models.

It employs twice as many parameters as the existing baselines. Additionally, the number of flops is also doubled and then multiplied by the number of iterations needed. This results in a model requiring significantly more computational resources than the baselines, to reach performance almost on par with RF and CtCAE.

Please note that In fact, our model has far fewer parameters compared to all competing baselines. For the models shown in Table 1, our model is much smaller than competing baselines by a factor between 6-23x. This further highlights the computational benefits of our simple ideas of complex-valued weights and recurrent computation.

Please find the model parameters for all models shown in Table 1 is listed below. Model parameters (rounded up to the nearest thousand) for all results shown in Table 1. Total number of model parameters for all models expressed in terms of number of real-valued floats.

Please find the wall-clock times (rounded up to the nearest minute) for training for our model and the best performing baseline model RF on Tetrominoes and dSprites to achieve the same results as shown in Table 1. Both models have been trained using the exact same hardware setup (1 NVIDIA P100 GPU).

From the above table we can see that SynCx converges much faster than RF on Tetrominoes where it also significantly outperforms RF in terms of ARI scores. On dSprites SynCx’s grouping performance is comparable to RF and both models take comparable times to converge. We thank the reviewer for bringing up these concerns. We hope that the additional information here resolves their concerns regarding the fairness of comparison of our model against baselines. We will highlight these points about our model and baselines in the updated version.

Are the baselines 100% comparable with SynCx? Are the baseline models still comparable in terms of number or parameters?

We believe that the comparison of SynCx with baselines is fair. Notably, our model has far fewer model parameters than competing baselines as shown above.

If not, how could the authors explain that the MSE loss of SynCx is always higher compared to RF and CtCAE (except on dSprites)?

Since our model has far smaller (weights) compared to the baselines and so it is reasonable to expect higher reconstruction loss for our model compared to RF and CtCAE. However, we would like to emphasize that the end goal of training this model is not to achieve better image reconstruction but learning better representations, i.e. phase synchronization towards objects.

end of part 3/3

The example in Figure 1 is used to motivate the need for recurrence and complex-valued weights; have the authors tried to evaluate their model on similar stimuli to confirm that it can group the objects accordingly?

Please note that this illustrative example was intended to essentially serve as a pedagogical tool to explain the essence of the idea and provide intuition to the readers. All our experimental benchmarking is done on standard datasets of interest to the object-centric learning community. We have not trained/tested our model on such specially constructed types of stimuli.

The recurrent dynamic is done only on the phase: at each iteration t, only the phase at t-1 is fed as an input, and not the magnitude (the magnitude remains the input image at each iteration). Have the authors tried a closed-loop recurrence where the whole complex output at iteration t-1 constitutes the input at iteration t? This solution would seem more elegant and more in line with neuroscience as it is unlikely that the phase and firing rate of each neuron can be split in such a way.

The process of grouping needs to be carried out on the input image. This grouping process iteratively refines a noisy initial phase map based on the features (magnitudes) and their associated bindings (phases). Analogously, if we see the iterative refinement process of the SlotAttention module, the recurrence is applied only on the slots and not on both the (slots, inputs). Purely, out of curiosity during early prototyping we did try the version the reviewer has described and found it did not work well. Lastly, we would like to reiterate that our design choices for architecture and training procedure are not intended to be biologically plausible.

The initial modReLU equation seems to include a normalization of the amplitude by z/|z|. Do the authors include it as well, and if yes, can they add this term to equation (2)? If not, can the authors explain why not include this term? Also, is there any constraint applied to the bias to prevent the amplitude from being negative? If not, do the authors make sure the amplitude remains positive or switch the phase value accordingly?

Please note the two ways of computing the modReLU activation are equivalent (see below). Therefore, this term has been included already by us.

Using the same notation across our paper and Arjovsky et. al, Given,
$y \in \mathbb{C}^d$
$| y | \in \mathbb{R}^d \rightarrow$ magnitude-component of complex-valued $y$
$\phi_y \in \mathbb{R}^d \rightarrow$ phase-component of complex-valued $y$
Our version (in equation 2) is written as,
$h = ReLU( |y| + b) * e^{i \phi_y}$
Original modReLU from Arjovsky et. al (equation 8 in their paper) can be written as,
$h = ReLU( |y| + b) * \dfrac{y}{|y|}$
$y = |y| * e^{i \phi_y}$ (Euler’s formula, as shown on the right side of equation 2 in our paper) Substituting this expression for y in the original modReLU we get,
$h = ReLU( |y| + b) * \dfrac{|y| * e^{i \phi_y} }{|y|}$ Canceling out the |y| terms in the numerator and denominator we get,
$h = ReLU( |y| + b) * e^{i \phi_y}$ (equal to our version of modReLU)

In short, the modReLU activation function ensures that the magnitude of the output always remains strictly non-negative. Further, since the ReLU function is only applied on the magnitude component of $y$, applying the activation preserves the phases across the pre-activation ($y$) and post-activation ($h$) complex-valued vectors (no flipping). Please note that the magnitude-component of output complex-valued activation $h$ is obtained by applying the modReLU activation function only on the magnitude component of $y$ ($|y|$). The magnitude component of $y$ ($|y|$) is strictly non-negative and the activation function is also strictly non-negative. Therefore the magnitude-component of $h$ ($|h|$) will always remain strictly non-negative.

Table 4 shows the benefit of von-Mises initialization compared to Uniform. Have the authors tried to initialize the phases at zero, as done in the CAE?

Please find the results using “zero” initialization for phases above. We can see that it performs comparably to the uniform case but worse than Von-Mises. This result is in line with our expectations. The best performing model variant (Von-Mises) samples “noisy” initial phases with an intermediate level of variance (Uniform -> very high variance, Zero -> no variance).

end of part 2/3

We thank the reviewer for their feedback on our work and appreciate the positive feedback on the quality of our presentation.

But this behavior can potentially also be obtained with real-valued weights. In fact, the weights are applied to the real and imaginary parts of a complex activity. The phase arrangement is therefore present in the activity, and can also play a role in the detection of such features.

Please note that simply using shared real-valued weights and applying it to compute matrix-vector products with real and imaginary parts of a complex-valued activation followed by summation alone does not lead to phase synchronization in the complex-valued activations in previous models (CAE++, CtCAE and RF). Phase synchronization in these models crucially relies on the use of an additional inductive bias i.e. the gating mechanism ($\chi$-binding) as noted by us (see lines 46-50) and echoed by Loewe et. al [23, 25]. Please refer to Table 2 in [23] and Appendix section D.3 in [25] where they perform an ablation on CAE and RF models without the $\chi$-binding mechanism. This ablated model variant fails to phase synchronize towards objects. Lastly, regarding our usage of the word “critical” we mean that the use of complex-valued weights is a natural choice to instantiate this function in our formulation of synchrony outlined (Introduction section). We do not mean that this is the only way to achieve phase synchronization behavior in general. We’re happy to adjust the corresponding text to avoid any misconceptions.

This behavior does not seem to align with neuroscience evidence, contrary to what the authors claim.

We would like to clarify that our work does not claim that the SynCx model design and training procedure are biologically plausible. Instead we motivate the use of complex-weights and recurrence from a computational perspective as follows.

Using real valued weights allow you to activate a hidden feature if two input features are aligned in phase, but you may wish to activate a hidden feature that indicates that the two input features should be grouped apart (e.g., 180 degrees out of phase). This is a principled computational motivation for using complex weights. As for whether complex valued weights have biological plausibility, it's easy to think of mechanisms that effectively behave like complex-valued weights. The critical mechanism is to detect features operating at different relative phases, which in the biological case, will correspond to different firing times. Dendritic tree connectivity can implement synchrony that is achieved at arbitrary phase lags. We don't emphasize this in our paper (and certainly can, if the reviewer would find it helpful) because we are focused on showing the computational improvements and elegance that results from complex-valued weights. We hope this resolves any misinterpretations of our motivations.

And for each model and dataset, do the object assignment and the phase map result from the same layer, or can they result from different layers even though they are from the same model and dataset?

Please note, that for grouping performance (ARI scores) of our model in all Tables (across all datasets) in the paper, we always cluster the phases from the penultimate decoder layer at the last iteration N. Similarly, for all phase map visualization we apply t-SNE to perform dimensionality reduction to the d-dimensional phases from the penultimate decoder layer. In Figure 4, we simply showed how these phases evolve across iterations. The visualization of phase maps (Figure 4 and associated text) is simply a qualitative tool for inspection. We do not use it to make any claims about the grouping performance of our model against baselines. We will edit the corresponding lines in text to make this point clear and avoid any confusion.

And why do the clustering metrics differ between object assignment (k-means) and phase map (t-SNE)?

Please note, that the visualizations of phase maps (Figure 4) are simply the 1D projections (dimensionality reduction done using t-SNE) of 64-D continuous phase vectors at each location in the spatial feature map. The visualization procedure does not apply any clustering procedure. The 1-d projected phase maps shown in Figure 4 are not similar to the discrete object segmentation masks (like columns 2, 4, 6 in Figure 3).

Slot-based models are mentioned in the related work but their performance on this benchmark is not included in the results.

Thank you for this suggestion, this was noted by another reviewer as well. We will add the following object discovery results of SlotAttention to Table 1 in the updated version to provide more context. Slot-based models do not show this color bias on these datasets tested here.

We will also discuss this performance gap between SlotAttention and state-of-the-art synchrony-based models in the limitations section.

Missing citation: even though modReLU was mentioned in Trabelsi et al, 2018; the modReLU was initially proposed by Arjovsky et al., 2015.

We thank the reviewer for this correction, we will update the citation as suggested.

The authors use the term "robustness" (L55) to express the excessive reliance of models on color cues. I'd suggest replacing this term as robustness represents something different in the DL literature.

We thank the reviewer for pointing this out. We will use another term to avoid any misinterpretation. We would like to ask if the reviewer has any suggestions for alternatives?

L100: parentheses missing after citation 24. We thank the reviewer for this correction and will update it.

end of part (1 / 3)

We thank the reviewer for their support of the motivation behind the use of complex weights and its rationale. We thank them for their support of the simplicity of our model which eliminates the need for contrastive training, $\chi$-binding and other heuristics.

I appreciate that the authors acknowledges that the model still relies on color cues to group pixels to a large degree, which is true to previous models such as RF, and its limitation in dealing with metallic or matte texture.

We thank the reviewer for the positive comments about the honesty in our evaluation of current state-of-the-art synchrony models. We would like to emphasize that our method (SynCx) relies far less on solely color cues compared to current state-of-the-art synchrony models such as RF (see lines 285-287).

Another limitation is that the detection of object relies on post-hoc clustering of phase. So it remains to be seen how it works when there are many objects, i.e., whether the space of phase is sufficient to separate different objects.

In our experiments we show that our model is capable of handling up to 5 or 6 objects (see Figure 8 row 3 and Figure 9 row 4). Note, that the phase space is still not fully covered indicating that it could potentially accommodate more objects. We will comment on this as well in the limitations section.

Regarding the post-hoc phase clustering, this ability of synchrony-based models to maintain a continuous phase representation for binding adaptable by gradient-based learning can be viewed as a strength compared to slot-based models where the binding information, i.e. number of slots and slot capacity are set apriori and not adaptable by gradient-based learning. This makes it difficult for slot-based models to adaptively change the coarseness of grouping at test-time without more fine-tuning or even re-training from scratch. Whereas our model (and synchrony-based models in general) maintain a continuous representation of object bindings adaptable by gradient-based learning. And we can post-hoc get different groupings with varying levels of granularity by modulating the clustering procedure applied on the continuous phase map. This gives the phase representation produced by synchrony-based models greater flexibility in chunking up the input image.

it is unlikely for solution proposed in the current model to be an ultimate solution for object perception. One may even ask whether the performance is mainly gained by the bottleneck or by the phase synchrony being proposed.

We appreciate the comments from the reviewer. We agree with the reviewer that we don’t believe that the contributions presented here are the ultimate solution and we have not made such claims in our work either. Instead, we advocate for a simpler approach (lesser inductive biases and simpler training procedure) compared to all other synchrony-based models (CAE, CAE++, CtCAE and RF).

it seems that a comparison with earlier models not in the synchrony-based binding family, such as MONet or slot-attention might provide a clue. But such comparison is unfortunately not provided.

Thank you for this suggestion. We acknowledge the gap in unsupervised grouping performance between current state-of-the-art synchrony-based models and slot-based approaches such as SlotAttention as noted by the reviewer. We will add the following object discovery results of SlotAttention to Table 1 in the updated version to provide more context.

It is noticeable how the performance gap between recent synchrony-based models (esp. SynCx) and SlotAttention has been narrowing on Tetrominoes and dSprites. We will also discuss this performance gap between SlotAttention and synchrony-based models on CLEVR dataset in the limitations section. Synchrony-based models have received far less attention by the research community and are making strides very recently to bridge the gap to predominant slot-based models which have been widely improved by the community efforts over a longer timeframe. Therefore, we view synchrony-based models as an exciting research direction with open problems to solve.

Although the invariance has been less focused by most of the object-centric models, it can be interesting to evaluate whether the phase is similar for similar object across images.

This is an interesting question, regarding representational invariance for an object across viewpoints. When using synchrony-based models, what matters for separating objects is the relative phase (phase difference) between object phases rather than the absolute phase values assigned to pixels of any particular object. In fact, it’s plausible that different objects could take the same phase value (eg. $\dfrac{\pi}{3}$) across different iterations or when starting with different random initializations for the phase map. In the reviewer’s example different views of the acorn are not encoded by some absolute phase value (eg. $\dfrac{\pi}{4}$) thereby leading to invariant representations of the acorn. Lastly, we believe that this approach of using multiple views of an object for prediction provides useful training signals for an object-centric model. However, this is outside the focus (scope) of the research questions addressed here and an interesting avenue for future work.

end of part 1/2

In the demo for dSprite in Fig 4, it appears that the phases are already well separated after the 1st iteration, but got more spread out in the second iteration and got back better in the third iteration. So I wonder whether such fluctuation in clustering is common.

We appreciate the reviewer’s feedback on our phase map visualization. Please note, these visualizations of phase maps (in Figure 4) are not the same as the discrete segmentation masks (clustered phases) shown in columns 2, 4, 6 of Figure 3. In simple terms, the visualization technique in Figure 4 is used to project a high-dimensional phase vector (at every spatial location) to 1D by applying dimensionality reduction (t-SNE). Our visualization technique for phase maps extends similar techniques provided in prior models (see Figure 4 in CAE [23]) to high-dimensional phase features from hidden layers. We cluster the phase maps from the last iteration to extract discrete segmentation masks for objects after the iterative process has settled to a relatively stable configuration. It is reasonable to expect that phase values will fluctuate (to some degree) before this process has settled as the phases are initialized with random (noisy) values.

end of part 2/2

We thank the reviewer for their positive comments regarding the simplicity of our model, the ease of understanding the rationale behind our design choices and for noting that the evaluations support the main claims in our paper.

It has been shown that TSNE have some limitations, specifically results can change dramatically based on the choice of iterations and perplexity. How were these values selected? Also perhaps good using another dimensionality reduction technique to perform the visualization such as UMAP.

We thank the reviewer for these helpful recommendations regarding our phase visualization process. Regarding the t-SNE hyperparameters used (listed in lines 564-566) we tried out a few values for n_iters=[250, 500, 1000] and perplexity=[15, 20, 30] in the prototyping phase. We settled on the values of n_iter=500 and perplexity=20 for the visualizations shown in the paper since they gave good initial results (visual inspection) within a reasonable amount of processing time per image. During the prototyping phase, we compared UMAP against t-SNE for performing dimensionality reduction in our visualization process (download PDF from https://file.io/gMEMsZWRKNZX). We used the default UMAP hyperparameters and t-SNE hyperparameters listed above for this comparison. We find that t-SNE produces qualitatively better projections than UMAP, i.e., phase clusters correlate better to the number of objects in the image (see panels 2,3 and 4). Lastly, the visualization process takes around 8 seconds per image with t-SNE compared to 24 seconds per image for UMAP. Therefore, we decided that t-SNE is the better choice for our use case here.

Maybe a minor question, in figure 4 there is a visualization of the iterative process, showing how every iteration seem to provide a better grouping result. However, in the phase step is not clear to me how every step is contributing to the propagation of the local context.

Thank you for your question. We agree with the reviewer that it is difficult to visually interpret from sequences of phase maps and radial plots to see how local context gets propagated. The illustrative example in Figure 1 was meant to serve as an idealized mode of operation of such a system. Regarding a way to visually show how local context is propagated. We think a possible approach would be to visualize/inspect local activation patches to show some measure of ‘agreement’ in their phases across successive iterations. Currently, we do not know a simple visualization method to visually show this phenomenon. We are curious to know if the reviewer has some recommendations for the same.

I want to thank the authors for the responses and the t-SNE experiments. It does seem that the UMAP has a harder time showing the separation of some objects. For instance, in the third one it may even lead to a failure mode. Perhaps a good point to add to the limitations. For the second question I wonder if a the difference between consecutive steps would provide the regions with higher change, and perhaps there is a connection there, but I agree with the authors that this may be hard to visualize in a simple manner. I will increase my score one point, but looking forward to discuss with the other reviewers and see how the other threads evolve.

We thank the reviewer for their positive comments and support for the simplicity and elegance of our model/training, quality of our experimental validation/analysis and the overall presentation. We’re happy that the reviewer enjoyed reading our work.

However, it is unclear how well SynCx would perform on scenes with overlapping objects that share similar low-level features (e.g., same color or shape). Specifically, would the phase representation effectively encode the presence of multiple distinct objects at overlapping pixels, mimicking human vision's ability to separate them?

This is an interesting observation about the phase representation. In Figure 9 (row 3) you can see two blue cubes (same shape / similar color) which partially occlude each other and our model has achieved good grouping of the two objects. Looking at the clustered phase maps (‘Mask’ column 4) the discretized phase at the overlapping regions encodes the presence of the object in the front and does not encode the occluded object as well. Looking at the t-SNE reduced phase map (‘Heatmap’ column 6) also we see that overlapping regions encodes the presence of the object in front (shades of blue) and does not encode the occluded object in the back (shades of purple).

Another example, in Figure 9 (row 1) you can see a yellow cylinder and purple cube both metallic (same texture) which partially occlude each other. Again we see the same behavior in the phase representation where in the overlapping regions has specialized towards the object in front rather than the occluded object at the back.

if the number of objects in the scene increases significantly during test time, would SynCx still perform effectively with the same number of iterations used during training, where the number of objects was lesser?

That’s an interesting question. We suspect that what matters more for convergence is the density of objects. Because of the convolutional architecture, patches are processed in parallel, and more objects won’t necessarily be harder to resolve if they are dispersed in the scene. We would like to investigate convergence time as a function of number and density of objects, but unfortunately the rebuttal period was too short to provide you with a concrete answer.

While SynCx shows promise, it is uncertain whether the approach would generalize to binding on naturalistic stimuli. This limitation is shared with existing baselines too, but addressing this gap in future work would be a valuable extension of SynCx.

We share the reviewers view on the scalability of this class of models to more visually complex scenes. We have shown the current limits of these synchrony-based models even on simple datasets like CLEVR which is far from perfect decomposition (ARI score of 1.0). We hope to continue research on these synchrony-based models to bridge this gap in future work. We view the contributions in this work as an initial template towards this end goal due to the simplicity of the model and its training procedure.

What happens if SynCx is run with more recurrent iterations than what was used during training? Is the model's binding encoding stable at RNN 𝑡>𝑡𝑡𝑟𝑎𝑖𝑛, or does the object discovery performance degrade during this "extrapolation" phase?

Table below shows the object discovery performance for our model on dSprites when trained with 3 iterations and evaluated using 4, 5 and 6 iterations. We can see that there is no degradation in object discovery performance of our model when we extrapolate the number of iterations used at test time.

We thank the reviewer for acknowledging the simplicity of our approach which does not require sophisticated heuristics or additional training objectives or forms of supervision. We address your specific concerns below.

L178 "We use latent-level complex-valued features ... than simply color cues (RGB space)": This seems contradictory to the authors primary claim. If only the phase information is used for the clustering analysis (and this is stateful) then it is unclear why the authors suggest the use of latents vs outputs? Can the authors clarify?

The short answer to your good question is that we tried to read out from the output phases but they were inferior to using latent phases to segment objects. We found that the richer latent-level phases associated with higher-order features are useful for grouping. We also experimented with variants of the model that used a single (shared) phase for all outputs, and where each output feature used its own phase. In the classic supervised learning approach to segmentation with complex-valued units (model trained to predict image and segmentation mask), we could cluster the output phases, but the training signal supported meaningful information in the output phases. We are continuing to try to understand why phase information is lost at the output in the fully unsupervised setting, but one reason is likely that a unidimensional phase representation is too restrictive. In support of this conjecture is the fact that, in RF, Lowe et. al [25] have incorporated multiple ‘phase’ (rotation) components for each neuron. Our alternative to this, is to use the high-dimensional phases from latent features. We will comment on these aspects in the updated version.

The color-lesion experiment on CLEVR has potential low-level confounds apart from color (like shadows, shading, etc.) that models can use as shortcuts for binding.

While it is true that there are these low-level differences in the image, this is far less of a cue than color signal. Colors can be accessed directly from the pixel values whereas these other cues require integration of multiple pixels. In short, the color cues are a shortcut feature that has high availability (Hermann et al. 2024) because individual pixels can specify the output grouping. Removing these cues (shortcut features) allows us to evaluate how much the other models rely on them (and to show that ours does to a lesser extent). Additionally, shadows make it perhaps even harder to do correct binding, and shading inside one object (e.g. for shiny metallic objects) arguably also makes the task harder.

On the Foundations of Shortcut Learning, Hermann et. al 2024.

Why did the authors have to retrain SynCx on the grayscale dataset? If binding-by-synchrony is faithfully implemented by their model, shouldn't this work zero shot? If this is not the case, then what is the rationale for it?

The experimental setup as suggested by the reviewer would introduce a sort of test-time out-of-distribution evaluation that challenges all object-centric models, as illustrated by the fact that SlotAttention [33] which performed similar grayscale experiments required retraining from scratch (see Figure 4 in [33]). Since the statistics of features belonging to any object have changed due to the grayscale transformation it is reasonable to expect the previously learned binding mechanisms on the colored images requires retraining the model weights. As such, it would be quite unreasonable to expect current state-of-the-art synchrony or slot-based approaches (eg. SlotAttention [33]) to zero-shot be able to group grayscale versions if it was only trained on RGB counterparts. Lastly, this experiment on grayscale CLEVR is designed to test whether the models have the capability of leveraging other image statistics other than color. And we show that our model (SynCx) significantly outperforms previous synchrony-based models.

The authors raise a point about instance-level groups being the focus of object-centric learning (L59). Given that, there are very few (maybe 2) examples presented where the scene contains multiple instances of the same object (same color, size, orientation, etc.) and the corresponding outputs from SynCx. Can the authors present these?

The point about instance-level groups was intended to highlight a commonly observed failure mode of state-of-the-art baseline model RF [25] which groups all instances of a semantic category such as ‘cow’, ‘train’, ‘bread slice’ or ‘banana’ as one group. This can be seen in Figures 2 & 20 in RF [25] where it groups different cows or bananas or trains as one entity. Note here, that the two cows or bananas or two bread slices are not exact copies (replicas) rather these object instances only partially share some features such as color, texture, shape as they belong to same semantic category. We can also see this failure mode of RF surfaces again on our datasets when it fails to disambiguate between two blue tetris blocks or two yellow cylinders in Figure 3. It primarily seems to use color as a shortcut feature to discriminate between groups on these datasets leading to systematic grouping errors. This failure mode of RF is exacerbated on Tetrominoes leading to very poor ARI scores (in Table 1).

In contrast, we see that our model is capable of disambiguating the separate entities in groups of two or three tetris blocks with same color and texture (row 2, 3 in Figure 7) or two hearts (row 2 in Figure 4 and row 1 in Figure 8) or three squares (row ) or two red balls (row 2 in Figure 9) or two blue cubes (row 3 in Figure 9).

end of part 1/3

It is understandable that the test loss is lower for the model without bottlenecks since it has a greater number of parameters.

Actually, the reviewer's claim is incorrect. The confusion is natural and our fault because ordinarily when discussing 'bottlenecks', spatial resolution and number of parameters are confounded. However, because we were concerned about this, our experiments actually ensure that both ‘SynCx’ and ‘SynCx w/o bottleneck’ have the same number of parameters. Instead we only vary the spatial resolution of feature maps. So the bottleneck is in spatial resolution but not in the number of parameters. We must be more explicit on this point in our paper; thanks to the reviewer for highlighting the natural confusion. We will adjust the corresponding text to communicate this better.

The importance of bottlenecks, in my opinion, originates due to the effective receptive field sizes in these layers and has nothing to do with complex-valued units or synchrony per se. It is unclear what this analysis adds to the manuscript.

Bottlenecks ( using convolution with stride > 1) in representational capacity are crucial for a fully convolutional autoencoder because they force it to learn a compressed representation of the input in the latent layers. To test the reviewer’s hypothesis about effective receptive field size being the mechanism that drives phase synchronization, we ran another control experiment where we increased the receptive field size by increasing the kernel size of the convolution filters in our model while using a stride of 1 (no bottlenecks).

The Table above shows results for our model which uses increasingly larger kernel sizes (3, 4, 5 and 6) and stride of 1 (no bottleneck). We can see that even when the kernel sizes are increased without having any bottlenecks it does not lead to increasingly better phase synchronization towards objects. The presence of representational bottlenecks (smaller spatial resolution of feature maps) using convolutions with stride > 1 leads to much better phase synchronization behavior in our models compared to the results shown above.

From the results presented, it seems like there is diminishing returns after just 2 iterations? If that's the case, then wouldn't this approach be very comparable to feedforward methods? Moreover, can the authors train SynCx on a large number of timesteps (say 10) and observe the dynamics of convergence in the phases during test time? Claims about "iterative constraint propagation" needs further backing. Perhaps these datasets are too easy?

Yes, we agree with the reviewer's hypothesis. As supporting evidence, we find that as the visual complexity of the data set increases, the number of iterations to converge to the best decomposition increases (e.g., 3 iterations for Tetrominoes and dSprites, but 4 for CLEVR). We were also surprised that convergence would be so quick and indeed, any recurrent constraint propagation algorithm can be implemented as a feedforward (unrolled) model. We describe the process as constraint propagation to provide an intuition about the nature of the computation. And we agree with the reviewer that as we move to even more complex data sets, the number of iterations will increase further. Lastly, please note that there is still a way to go for these models to achieve perfect grouping scores ARI: 1.0 on these datasets especially on CLEVR so we believe the set of datasets picked are an appropriately difficult benchmark suite to test these synchrony-based models.

I found it hard to grasp the take away of this section. The Von-Mises distribution (with mean 0, and concentration 1) biases most of the units in the initialization to have similar phases. The random initialization performed pretty poorly. What does this suggest though computationally?

This suggests that it is important to tune the variance of the distribution used to sample the “noisy” phase map. The Von-Mises distribution that we use is simply the circular analogue of the normal distribution which is a reasonable choice. This section provides practical recommendations for initialization schemes for the initial “noisy” phase map. Since the adoption and training of complex-valued autoencoders is still largely underexplored (non-standard) in the deep learning research community compared to the default deep learning models that use real-valued weights/activations. We believe it’s important to detail the key engineering choices to facilitate future users to train such autoencoders well on new benchmarks.

end of part 2/3

Figure 2 is slightly misleading in its depiction. The magnitude of the output \mu_z^1 is shown to be the same as \mu_x. This derails the reader from understanding the flow of information across timesteps. Notationally, denoting time as a superscript and outputs as z’s is confusing from a reader’s perspective. A clearer version could be something along the lines of \hat{x}(t) = \mu_{\hat{x}}(t) \odot \exp{i \phi_{\hat{x}} (t)}

We thank the reviewer for the suggestions for improving Figure 2. The model diagram (and the corresponding input/output images) are meant to aid reviewers' understanding of the model and are therefore an approximation of the process. We will replace the output magnitude image with a reconstructed image instead. We thank the reviewers’ suggestion for a revised notation for the complex input $x(t)$. Please note, the magnitude component is independent of the iteration and is always clamped to the input image ($\mu_x$). Accordingly, we can rewrite the expression as $x(n) = \mu_x * e^{i \phi_{\hat{x}(n)}}$. Lastly, we apologize for the typo in line 159: clamped to the image ($x'$) -> clamped to the image ($\mu_x$).

Minor (typos): We thank the reviewer for these corrections and will revise them.

end of part 3/3