Thank you very much for your thorough review and constructive feedback.

Q1: Presentation/Clarity/Rigor

R1: We sincerely apologize for the confusion caused in the introduction and methods sections. Our aim was to bridge concepts from neuroscience with controllable generation in AI, hoping to stimulate cross-disciplinary inspiration and collaboration. However, in attempting to reach audiences from both fields, we presented the methods in a coarse-grained manner, leading to misunderstandings among readers, including reviewers. To address this, we will: 1）Rephrase: Clarify terms, sentences, equations, and figure captions in sections 1 and 3 to ensure their accuracy and accessibility. 2）Enhance understanding: Provide a more detailed diagram of the proposed modular autoencoder in Section 3 (see it in PDF Fig. R4), including a clearer representation of the prediction matrix M, equivariant loss, and symmetry loss. Refer to GR2 for more.

Q2: Impact

R2:

1. This proposed method’s most significant contribution lies in introducing a novel self-supervised conditional generation training framework. This framework enables the acquisition of broad capabilities through a self-supervised approach, offering the potential to fully leverage the scaling law and achieve a large-scale foundation model for controllable generation. Furthermore, by fine-tuning the foundation model on specific downstream tasks or utilizing adapters, powerful specialized models can be easily derived.
2. Since there has been extensive research on conditional generation, we focus on designing a modular autoencoder based on proposed equivariant constraints, which is crucial for subsequent self-supervised controllable generation. And the visualization of the autoencoder component and the demonstration of its conditional generation capability on various tasks with zero-shot generalization prove the effectiveness of our method.
3. On specific tasks, our method still falls short of the generative capabilities of dedicated supervised methods. A major reason for this is the use of labeled data, such as sketch-image pairs, in supervised methods. If we were to fine-tune our model with labeled data, its performance on specific tasks would significantly improve.

Our primary goal at this stage is to demonstrate that the proposed self-supervised framework can emerge broad conditional generative capabilities. Further supervised fine-tuning will be explored in future work. Refer to GR1 and GR4 for more.

Q3: Variability and Comparison with principal components.

R3:

1. In PDF Fig. R5, we demonstrate the ability to precisely control the performance of the generated image on specific components by adjusting the intensity of the components (specifically, by multiplying different coefficients). It is evident that our method can easily manipulate saturation and contrast by changing the coefficient of HC0 and HC1.
2. Regarding the principal components, in PDF Fig. R4, we showcase the principal components obtained through PCA. A fundamental difference between components of PCA and modules of our method is that the different components in PCA are independent. As a result, PCA lacks the ability to decouple highly correlated internal representations into relatively independent and complementary representation modules, which is the core of zero-shot conditional generation capability.

Q4：Minors

R4:

1-2）33-36 line rephrase: We revise it to "The brain’s remarkable ability to generate associations emerges spontaneously and likely stems from two key mechanisms. One is the brain’s modularity in terms of structure and function, enabling it to decouple and separate external stimuli into distinct features at various scales. This modularity is essential for a range of subsequent cognitive functions, including associative generation."

3-7）We will fix them.

8）We will unify z and f to z.

9）We will revise z(I) to z.

10）$\delta$'s range: $\delta$ contains 3 parameters: two translation parameters $t_x$, $t_y$, and one rotation parameter $\theta$ (see PDF Fig. R4). The range of t_x and t_y is [-0.5s, 0.5s), where s is the convolution stride. The range of theta is [0, 2pi). Each parameter or dimension is sampled at n (i.e., n=24) equally spaced values.

11）Refer to GR2.

12-14）We will revise them.

15）Refer to R1 and GR2 .

16,17）we will fix it.

Q5: Questions

R5:

1）Yes, we will revise it to "z".

2）The symmetry loss aims to further enhance the internal correlation (or symmetry) within modules, building upon the equivariant loss. Without symmetry loss, it will learn unrelated features within module as shown in PDF Fig. R3, referring to GR2.

3）We used random translation and rotation augmentation within 0.3 times the image length and 360 degrees. When removing translation augmentation or rotation augmentation (PDF Fig. R2), modular features are still learned. The difference is that without translation augmentation, the learned features lack orientation selectivity and resemble Gaussian difference, while without rotation augmentation, features within each module have the same orientation.

4）We train it for 40000 steps with AdamW optimizer on a cosine anneal strategy with a start learning rate of 0.005.

5）Yes, it’s a single-layer or can be equivalently considered as a single-layer network.

6）We use plattform https://www.wjx.cn/ to collect subjective evaluations, gathering 40 questionnaires. Each questionnaire contained 12 pairs of comparison images and one original image. Participants were asked which image, out of the two, had better fidelity and aesthetics basing on the original image. （see appendix A.4.4 in main text）

Q6: Limitation

R6: Refer to R2.

Thank you very much for your efforts and valuable feedback on our paper.

Q1：What are we learning from pattern completion?

R1：Pattern completion focuses on the relationships between different module features at a global scale. By learning these relationships, SCG can utilize information from a subset of modules as clues to complete or generate information in other modules, thereby achieving various conditional generation tasks. Refer to GR3.2 for more.

Q2：Section 3 should be written more clearly.

R2：We sincerely apologize for the lack of clarity in some of our expressions. We will further clarify the captions of Figure 2, the prediction matrix M, the distribution of $\delta$, equation (6) and equation (8). Since we only presented a very coarse-grained overview of our framework in Figure 1, it may have been difficult to grasp the finer details of the method, especially regarding the modular autoencoder part. To address this, we have added a more detailed structural diagram of the modular autoencoder in PDF Fig. R4. It provides a more intuitive visualization of the modular autoencoder with proposed equivariance constraint, including the prediction matrix M, the equivariant loss, and the symmetry loss. And we will add this figure in section 3. Also can refer to GR2.1.

Q3：It appears that the proposed method is more of an image reconstruction method... The proposed SCG method would not be able to generate an image based on my sketch. Is this correct?

R3：The proposed SCG can generalize to both reconstruction-oriented tasks and generation-oriented tasks, such as sketch (main text Figure 5), LineArt (PDF Fig. R2), and ancient graffiti (main text Figure 7) in a zero-shot manner.

1. In the PDF Fig. R2, we use SCG to generate an image of “a monkey climbing on the moon” using a sketch as control.
2. Different modules have different characteristics. For instance, HC0 extracts color information, while HC1 extracts brightness information. This makes them more suitable for reconstruction-oriented tasks like image super-resolution (PDF Fig. R2), dehazing (PDF Fig. R2), and colorization (main text Figure 6 and Appendix Fig. S8).
3. For HC2 and HC3, the extracted information is more abstract, focusing on edges. Thus, they are more suitable for generation-oriented tasks such as conditional generation under sketch (main text Fig. 5), line art (PDF Fig. R2), and ancient graffiti (main text Fig. 7).

It is undeniable that our fully self-supervised controllable generation approach was not specifically trained for tasks like sketch and line drawing, and therefore, its performance and stability fall short compared to dedicated supervised methods. However, SCG’s performance on specific tasks can be significantly improved by supervised fine-tuning for those tasks, such as adding sketch and images pairs for training. We will incorporate this into the discussion of future work. Refer to GR4.

Q4: Would a different representation to condition on have worked better than Canny?

R4:

1. We have introduced controllable generation using depth maps, normal direction maps, and semantic segmentation maps as conditions for comparison (see PDF Fig. R1). These conditions are more abstract than Canny edges, offering a larger generation space. However, they require a supervised pre-trained network to extract the conditional information. Due to their abstract nature, the generated images exhibit greater diversity and aesthetic appeal.
2. However, they are highly sensitive to the distribution of the conditional image. For instance, in tasks involving generating ancient graffiti and ink paintings (PDF Fig. R1), most condition extractors fail to extract the correct conditional information, leading to uncontrollable generation results. Conversely, when accurate conditional information is available, such as high-quality depth information in ink painting, the generation results surpass those of SCG and the Canny operator. For further details, please refer to GR4.1.

Q5: What kind of GPU and how much RAM is required to run their code.

R5: We use A100 GPU and it requires about 11G RAM to run it per image.

Q6: More detail in the caption of Figure 2, especially for part C of the figure.

R6: Revised caption of Figure 2: "Feature Visualization of modular autoencoder. Each panel shows all features learned by an individual model with multiple modules (one module each row). We trained modular autoencoder with a translation-rotation equivariance constraint on a)MNIST and b)ImageNet, respectively. c) On ImageNet, We also train an autoencoder with an additional translation equivariance constraint besides the translation-rotation equivariance constraint. d) We visualize reconstructed images by features of each module in c."

Q7: More details on $M^{(I)}(\delta)$,distribution of all $\delta$, Eq. (6) and Eq. (8).

R7:

1. $M^{(i)}$ is a codebook of learnable prediction matrices that can be indexed by $\delta$. $M^{(i)}$ is a 3D codebook, with the three dimensions corresponding to the three parameters of $\delta$: two translation parameters $t_x$, $t_y$, and one rotation parameter $\theta$ (see PDF Fig. R4).
2. The range of $t_x$ and $t_y$ is [-0.5s, 0.5s), where s is the convolution stride. The range of $\theta$ is [0, 2pi). Each parameter or dimension is sampled at n (i.e., n=24) equally spaced values, which is also the range of sum in Equ (6). The prediction matrix $M^{(i)}(\delta)$ is obtained from the codebook through linear interpolation.
3. Ideally, C in Equ (8) reconstructs the complete representation from the partial module $z^{(i)}$ and its index i. In practice, the training process of C aims to achieve this goal, for example, using a diffusion model-based conditional generation process. Refer to GR2 for more.

Q8: Some typos and imprecise term.

R8: Thank you for your careful review and pointing out these issues. We will fix them.

We are grateful for your fondness of our idea and are truly encouraged by it.

Q1：Evaluation may lead to a serious unfair comparison because the original ControlNet is not trained on MS-COCO.

R1：The ControlNet we used as baseline is trained from scratch with the same setting as the proposed SCG. We apologize for the inconvenience and confusion caused to readers, including reviewers, by placing the introduction of some of our training settings, including the comparison baseline ControlNet, in Appendix A.4.2 of the main text. We will move this information to the main text’s experimental section in the future.

Q2：Further experiments are conducted, such as whether using it as pre-trained weights for further fine-tuning on labeled tasks such as depth, segmentation, etc. will bring further improvements of controllability? The zero-shot capability alone cannot fully demonstrate the effectiveness and necessity of the proposed self-supervised training.

R2：This is an incredibly insightful suggestion, opening up new avenues for our research. Previously, we focused on fully unsupervised methods for modular, independent, and complementary feature disentanglement, exploring the emergent capabilities. Your suggestion has broadened our perspective. While fully self-supervised training can exhibit impressive emergent capabilities and demonstrate strong generalization across data distributions and task variations, it still falls short compared to supervised methods in specific tasks. Leveraging our self-supervised model as a pre-trained model and then fine-tuning it on labeled data for specific downstream tasks, such as sketch-conditioned generation, super-resolution, or dehazing, can significantly improve performance on those tasks. We believe this approach will enable both competitive capabilities on specific tasks and the ability to generalize to out-of-distribution data and tasks. Moreover, this fully unsupervised pre-training process has the potential to replicate the scaling law observed in large language models and other domains, establishing a foundation model for conditional generation. We will incorporate this into our future work section. Refer to GR1.1 for more.

Q3：Ablation experiments on the number of modules (from modular vae) and the types of Equivariant Constraints.

R3：We apologize for our focus on demonstrating the zero-shot generalization capability of the conditional generation model and neglecting this aspect. We have included an ablation study in PDF Fig. R3.

1. As visualized in Fig. R3, when the number of modules or the number of convolutional kernels within each module varies, the modular autoencoder network can still reliably produce relatively independent and complementary functional specializations.
2. When translational motion is removed, modular features are still emerged, but they lose their orientation selectivity, exhibiting a center-surround receptive field similar to a Difference of Gaussians. When rotational motion is removed, features within each module exhibit the same orientation selectivity.
3. When the symmetry constraint is removed, modular features are still generally produced, but some modules may contain multiple unrelated (or asymmetric) sub-features. As shown in Appendix Fig. S1 a and d of the main text, when the equivariant constraint is removed, the model cannot generate relatively independent and complementary functional modules.

We will add these ablation experiments to the appendix. Refer to GR2.2 for more.

Q4：Does SCG also have certain zero-shot capabilities under some other conditions such as Edge/LineArt/Segmentation?

R4：Yes.

1. We quickly test SCG in other zero-shot tasks including generation conditioned by LineArt (see PDF Fig. R2), as well as super-resolution (see PDF Fig. R2) and dehazing (see PDF Fig. R2)(Refer to GR4.2 for more).
2. Furthermore, we also can manipulate saturation and contrast of the generated image by changing the coefficient of HC0 and HC1, see in PDF Fig. R5. (Refer to GR4.3 for more).

This demonstrates the advantage of self-supervision over supervised learning, allowing the emergence of broader capabilities rather than just specific ones. It also indicates the effectiveness of the features learned by our modular autoencoder.

Thank you very much for your affirmation of our originality!

Suggestion：Better use "controllable generation" rather than “pattern completion” .

Response：Thank you for your suggestion. “Controllable generation” is indeed a more accurate and understandable term in the field of AI, as it clearly reflects the concept of generating content with specific controls. We will clarify this by explaining that “pattern completion” corresponds to “controllable generation” when we introduce the concept in our paper. From that point forward, we will consistently use “controllable generation” throughout the paper.

Q1：Quility: the evaluation is limited.

R1：We added comparisons to other conditions besides Canny, including depth maps, normal directions, and semantic segmentation in PDF Fig. R1, with each condition extracted by a pre-trained model. These condition extraction networks are highly sensitive to data distributions. For instance, all models fail to extract condition information for the ancient graffiti, resulting in low-fidelity generated images. When suitable features can be extracted, more abstract control conditions can achieve better generation results, such as ControlNet based on Canny and depth. Here are two possible reasons for this: 1) Our SCG was only trained on COCO, not a larger dataset; 2) SCG training did not use supervised data, such as sketch and image pairs. Addressing these two issues could significantly improve SCG’s ability on specific tasks. Our goal in this work is more focused on demonstrating that SCG can spontaneously emerge conditional generation capabilities on a wide range of tasks. Refer to GR4.1 for additional information.

Q2：Significance: suggestions to test on reconstruction task other than generation.

R2：Thanks for good suggestions. We quickly conducted simple tests on image super-resolution and dehazing tasks (see PDF Fig. R2) and still see that the proposed SCG has emerged with both super-resolution and dehazing capabilities. As SCG is fully self-supervised, its performance is hard to compare with supervised and dedicated networks. However, it has the potential to be used as a self-supervised pre-trained model and then fine-tuned on specific tasks, thereby enhancing the network’s ability in specific tasks while maintaining high generalization capabilities. Refer to GR4.2 and GR1.1 for additional information.

Q3：More analysis on symmetry loss vs equivariance loss.

R3：The equivariance loss is central to the modular autoencoder architecture, playing a crucial role in promoting independence between modules while simultaneously strengthening feature correlations within each module. The symmetry loss further enhances pairwise correlations (or symmetry) between features within a module, effectively suppressing the emergence of multiple unrelated sub-features within the same module. Removing the equivariance loss prevents the network from learning brain-like Gabor-shaped features and hinders the emergence of complementary functional specialized modules (see Appendix Fig. S1 a and d in the main text). While removing the symmetry loss allows the autoencoder to still generate complementary functional specializations overall, the correlations (or symmetry) within individual modules are reduced. This can lead to multiple unrelated features appearing within some modules (see PDF Fig. R3). Refer to GR2 for more.

Q4：What are the equivariance properties of HC1 and HC3 and make them useful for controllable generation?

R4：

1. It is noteworthy that all modules within the same network are under the same type of equivariant constraint (i.e., translation and rotation). The distinct functions that emerge from these different modules are entirely self-emergent and not a result of imposing distinct constraints on each module.
2. Modular autoencoder, under equivariance constraints, disentangles and decomposes into modular and complementary features. For example, HC0 represents color features, HC1 represents brightness features, HC2 and HC3 represent edge features, and HC4 and HC5 represent higher-frequency edge features. When we retain the brightness features (HC1) and drop the other features, the generation model will use the brightness features as clues to complete the other missing information, such as color. Therefore, conditional generation based on HC1 can be used for tasks like colorizing ink paintings or other re-color tasks. Similarly, when we retain the edge features (e.g., HC3) and discard other features, the generation module can use the edge information to generate color, brightness, and other information. Therefore, it often performs better in more abstract conditional generation tasks, such as sketch and ancient graffiti. For HC0, which represents color information, it is insensitive to fog due to its white color. Therefore, we can utilize HC0 for dehazing.
3. Different conditional generation tasks can be viewed as scenarios where certain aspects of an image are missing or corrupted. The goal is to use the remaining information as clues to complete or generate the missing information. The modularity of the autoencoder, resulting in relatively independent (at a local scale) and complementary features, allows for zero-shot generalization across a wide range of tasks, despite not being specifically trained for any particular task. Refer to GR3 for more.

Q5：Limitations: what useful features are NOT learned by this approach and why.

R5: While more features, such as depth (parallax), motion(e.g., optical flow), and semantic-oriented contours and instance segmentation, remain unlearned, they hold significant potential to further enhance the Controllability. Our current version of the autoencoder is limited to learning static, local features, thus preventing it from disentangling dynamic, volumetric, and semantically related modular features. We will discuss this in more detail in the Limitation section.