AdanCA: Neural Cellular Automata As Adaptors For More Robust Vision Transformer

We appreciate your valuable comments and positive feedback. We now address your questions below.

• Comparison with ViTCA

Please refer to the global rebuttal ”AdaNCA and ViTCA”.

• The difference between MLP processing concatenated channel vectors and Dynamic Interaction

Please see the global rebuttal ”The usefulness of Dynamic Interaction”.

• Integrating AdaNCA into pre-trained ViT models

Please see the global rebuttal "Integrating AdaNCA into pre-trained ViT models".

• Noisy training of baseline ViT models

We are keen to explore the possibilities of integrating NCA training strategies into the current ViT training pipeline, as we believe there are connections between the two model families (Section 3.1.1). Nevertheless, we want to point out that our focus is the integration of NCA and ViT. We made a first attempt at incorporating one training strategy, the stochastic update as you indicated, into current ViTs.

The stochastic update improves the model's robustness while undermining its clean performance. We assume this is because of the incompatibility between NCA which uses a local recurrent interaction scheme while ViT uses a global single-pass one. Despite this, it supports the notion that stochasticity during training indeed improves robustness.

• Magnitude of performance

We wish to highlight that AdaNCA can contribute to over 10% absolute improvement in adversarial robustness, as shown in Table 1, Section 4 in our main manuscript, with a small cost. For instance, ConViT-B-AdaNCA uses merely 3% more parameters and 7% more FLOPS to achieve more than 10x improvement compared to the enlarged baseline model (* sign). More importantly, our method achieves on-par or better results than a recent advanced modification of ViTs, namely the TAPADL method. Therefore, AdaNCA provides a good trade-off between the performance and computational costs, facilitating its deployment.

• Clarification

-- Line 163, NCA queries cell states:

We are happy to clarify more details of the NCA evolution. Following Equation 6 in Section 3.1, NCA evolves in a recurrent manner for certain steps. Instead of evolving for a fixed number of steps, NCA typically runs for random steps, where the number is randomly selected from a range. After evolution, the cell states are used for downstream tasks. In our case, AdaNCA outputs activations for the next ViT layer. Hence, the usage of the cell states happens at a random step, and we refer to this process as NCA querying the cell states. We will improve the text and make it clearer.

-- Readability of Table 1:

Thank you for your suggestion. We will improve the readability of Table 1 by adding arrows indicating the direction of performance growth. We clarify that the IM-C uses a different metric than other benchmarks that use accuracy, as stated in L257-258. It corresponds to a relative classification error compared to a pre-trained AlexNet, and thus lower is better.

-- Single Best Model:

Yes, our model performance is from a single best trained model, to facilitate the release of the pre-trained weights. The best model we obtained almost all fall into the last 10 epochs of training, with some exceptions in our ablation studies. All models are from the last 15 epochs of training.

We thank you for your valuable feedback on our experiments. We will improve the clarity of the sentence in the abstract. We now address your major concerns below.

• Choice of baselines

We fully agree with you that the original results reported in the TAPADL paper should be included for completeness, and we will add corresponding results to our manuscript. However, we'd like to clarify that we choose the original TAPADL models, instead of the one without ADL loss, and download them from the official code repository. We test them on our own machine, which might lead to different results from the original paper due to the difference between hardware and platforms. Importantly, the test results of our own code match the ones produced by their official code, which are the results reported in our paper. We do not train the TAPADL models as our focus is on the improvement of AdaNCA-enhanced ViTs compared with the baselines. We include the results from TAPADL models to showcase the strong robustness improvement that AdaNCA can achieve. In fact, our reported results match most of the reported ones in their main paper (citation [22] in our paper), with differences only in the IM-A results of both TAPADL models and the IM-R results of TAPADL-FAN. We assume these differences might originate from different data processing methods. However, we emphasize that we test our AdaNCA-enhanced model using the same setting as we test the TAPADL models, which follow the hyperparameters of the corresponding baseline models.

• Table 2 comparison with TAPADL

We'd like to point out that we do not have access to the official RVT model trained on ImageNet1K, hence we use the TAPADL model as a proxy in non-adversarial robustness evaluation. We state this compromise in Section C.8 in the Appendix. We will add this clarification to our main manuscript. In all other similar comparisons, as provided in Table 13 in the Appendix, we use the standard baselines.

• The usage of ADL loss

We clarify that we do not use ADL loss to train our models since we focus on our proposed architectural changes, as stated in L235-237.

• NCA parameters

We highlight that AdaNCA works with cells that have much higher dimensionalities. For example, NCA in texture synthesis typically adopts a cell dimensionality of 16. ViTCA operates in 128-dim space. In ViTs, however, the cell dimensionality can be 768 or 1024, resulting in a large amount of parameters in the MLP of NCA. We choose the hidden dimensionality of the MLP to be equal to the input dimensionality, as we do not want to have an information bottleneck (Section C.3 in the Appendix). Hence, an NCA working with a cell dimensionality of 1024 can have 1024 x 1024 x 2 $\approx$ 2M parameters in the MLP part. Moreover, we insert 2-3 AdaNCA models into ViT to achieve a good balance between the computational costs and performance improvement. Therefore, the final increase in parameters can fall in the range of 1-2.5M, as you indicated. Regarding the concatenation scheme, NCA parameters can exceed 10M as shown in the Global rebuttal "The usefulness of Dynamic Interaction". It is because the input dimensionality to the MLP is $\mathcal{M}$x larger if we use $\mathcal{M}$ kernels for interaction. Considering the above example where the cell dimensionality is 1024, the MLP now has the weight tensor of the first MLP being 4096x4096 with $\mathcal{M}=4$ if avoiding information bottleneck (Section C.3 Appendix), which is already 16M. We believe more efficient instantiation of the Update stage in NCA is worth exploring, and we are happy to add a paragraph discussing this future direction in our paper.

• NCA steps

We agree that our choice of NCA steps differs from all previous NCA models. We make this compromise as the computational costs increase linearly as the NCA step grows. We aim to minimize the increase in the number of parameters and FLOPS for scalability and we do not want the source of improvement to merely stem from the increase in the size and computation of the models. We show that more NCA steps will indeed contribute to the model's performance, as shown in the following table, while it introduces extra FLOPS. The model setting is the same as in our ablation study in Section 4.3.

We want to underscore that it is the architecture and evolution scheme that defines an NCA as introduced in Section 3.1, instead of the number of its recurrent steps. Admittedly, less steps will lead to a coarser path to the target state, and can potentially undermine the model's performance. Notably, with our scheme, AdaNCA has achieved generally strong robustness improvements compared to the baselines. It indicates that our choice of the recurrent steps of AdaNCA is a good balance between computational costs and performance.

• Choice of hyperparameters

Our choice of the training hyperparameters follows the settings of corresponding baseline models, as listed in Section C.6, L718-719. Hence, their differences in hyperparameters result in the differences in our experiments. We want to highlight that our AdaNCA can adapt to different training hyperparameters, e.g., learning rates, EMA (on/off, different decay rates), batch size, etc, which indicate the strong adaptability of AdaNCA in different training settings as well as its promise in combining it with any ViT models.

• Code release

Yes, we will release cleaned and well-documented code in case of acceptance, as well as the pre-trained models.

We thank you for your positive feedback on our work. We now address your questions.

• The generalizability and scalability of AdaNCA

We fully agree with you that applying AdaNCA in other computer vision tasks and scaling it to larger datasets and higher-resolution images are worthwhile. Given the fact that AdaNCA can scale up to much larger sizes than all previous NCA models and its effectiveness in large-scale image classification tasks, we are confident in AdaNCA's competitiveness in tasks such as robust semantic segmentation. In fact, we are actively looking into robustness benchmarks such as CityScape-C [1] and ACDC [2] and we are keen to explore these problems in the future.

• Elaboration of Dynamic Interaction

We are happy to explain our proposed Dynamic Interaction in more detail. Assume we have a token state $\mathbf{S} \in \mathbb{R}^{H \times W \times C}$. After each $C$-dim token interacts with its neighbors using $\mathcal{M}$ different depth-wise convolutions, we have $\mathcal{M}$ tensors with the same shape as the input token state $\mathbf{S}$. Our Dynamic Interaction first computes a token-wise scalar weight tensor $\mathbf{W}_{\mathcal{I}m} \in \mathbb{R}^{H \times W \times 1}$ using a two-layer CNN for each interaction results. Hence, the output of the CNN is of shape $H \times W \times \mathcal{M}$. The input of the CNN is the token map. The CNN consists of two 3x3 convolution layers with a batch normalization layer in the middle. The interaction results are then computed using the right-most hand side of Equation 8, in which a weighted sum is performed on the $\mathcal{M}$ interaction results. Each token has a different set of weights for aggregating the interaction results. Hence, the tokens dynamically adjust the weights of the combination on the interaction results based on both their own state and the states of their neighbors, which we term Dynamic Interaction. We will improve the texts of Section 3.2.1 to make it clearer.

• Novelty of Dynamic Interaction

Please see the global rebuttal ”The usefulness of Dynamic Interaction”.

• Developing patterns of AdaNCA

Please refer to Figure 1 in the R-PDF.

[1] Michaelis, Claudio, et al. "Benchmarking robustness in object detection: Autonomous driving when winter is coming." arXiv preprint arXiv:1907.07484 (2019).

[2] Sakaridis, Christos, Dengxin Dai, and Luc Van Gool. "ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

We thank you for the valuable and detailed suggestions as well as the acknowledgment of the originality, significance, and novelty of our work. We now address your questions below.

• Importance of recurrent updates

We fully agree with you that the recurrent update scheme of NCA is crucial for the model to be robust. We will modify L40-41 and L165-166 to include this point. We want to clarify that we do not ignore the recurrence but mention it in L37-38 to state the intuition behind the advantage of using NCA, and also in our ablation study (Section 4.3) on Recur.

• Comparison with ViTCA

Please refer to the global rebuttal "AdaNCA and ViTCA".

• Difference between AdaNCA and other NCA

We agree that a discussion on this topic can help readers better understand our method, including the training, hidden state usage, initial cell states, and the pooling strategy. As we do not include them in AdaNCA training, we will add them to the Appendix.

• Stochasticity in AdaNCA

We'd like to highlight that stochasticity during testing can hinder the evaluation of adversarial robustness by producing obfuscated gradients ([3] in the main paper), leading to the circumvention of adversarial attacks (L244-245 and Section C.7.1 in the Appendix). Table 5 in the R-PDF illustrates this issue, where we test the classification accuracy under CW attack. Stochasticity also results in inconsistent outputs. This is problematic for practical image classification tasks where reliable output is critical, unlike applications focusing on visual effects (Self-Organizing Textures (SOT), ViTCA) or collective behaviors (Self-Classifying MNIST (SCM)). The change in Clean Acc. in Table 5 indicates that given the same image, the stochasticity can lead to different decisions, hindering the deployment of the trained models in real-world scenarios. We also conducted your suggested experiment by removing the 1/p during training (Table 6, R-PDF). The drop in Clean Acc. and robustness suggests that downstream ViT layers struggle with varying NCA output magnitudes. Our dropout-like compensation scheme improves AdaNCA’s performance, which might be due to the different application scenario compared to previous NCA works.

• The usefulness of Dynamic Interaction and its multi-scale counterpart

First, we clarify that W_I is shared across scales.

-- Comparison with the concatenation scheme

Please see the global rebuttal "The usefulness of Dynamic Interaction".

-- Comments on other convolutions

Pros: Both spatial and 1x1 conv incorporate channel mixing in the interaction stage, potentially improving the model capacity of NCA.

Cons: Adding channel mixing can cause a drastic increase in the number of parameters and FLOPS, hindering scalability. For example, using spatial conv will result in ~10M more parameters when the token is 384-dim and the number of kernels is 4 (the conv will be 1536x384x3x3 for a single scale, and we use two scales). It will also modify the paradigm used in ViT and NCA. That is, to separate token mixing from channel mixing. The modification can potentially lead to redundant information.

-- The usefulness of multi-scale Dynamic Interaction

Our motivation for using multi-scale interaction is to perform more efficient token interaction learning over local scales since ViT already contains global information. Enlarging the neighborhood size will: 1) Complicate the process of selecting the neighbors to interact with; 2) Introduce excessive noise, making it difficult for tokens to accurately acquire neighbor information. 3) Repeatedly acquire the global information provided by ViT. The recurrence further amplifies the noise. We point out that ViTCA also suffers from the neighborhood size issue in image generation (Table 5 in their Appendix). Theoretically, self-attention can discount far-away information as the tokens also perform a weighted sum on the neighborhood values with the weight generated by itself while it struggles with large neighborhood sizes. In contrast, our model gains performance when S=2 (5x5 neighborhood), indicating the usefulness of our multi-scale module. Although increasing model capacity (as you suggested by using larger filters or increasing the receptive field of W_Ms) can improve the performance (Table 7, R-PDF), AdaNCA struggles with overly large neighborhoods. We will add a paragraph to discuss this issue. We note, however, that our work pioneers the application of multi-scale interaction in large-scale image classification tasks.

• Integrating AdaNCA into pre-trained ViT

Please see the global rebuttal "Integrating AdaNCA into pre-trained ViT models".

• Comment on the ablation study

Due to the space limit of the main paper, we did not include the analysis of the ablation study. Here, we provide our analysis below and we plan to add it as a Section in the Appendix: As shown in Table 3 (main paper), the highest robustness improvement is achieved with all components. Among them, turning off the recurrence leads to the largest drop in the robustness, as recurrence allows the model to explore more cell states than finishing the update in a single step. While it achieves the highest clean accuracy, it uses ~4x more parameters than our method, and the improvement of the clean performance is likely due to more parameters. Without any of the two sources of randomness, stochastic update and random steps, the model cannot adapt to the variability of the inputs and thus exhibits vulnerabilities against adversarial attacks. Finally, turning off our Dynamic Interaction will cause drops in both clean accuracy and robustness, as tokens cannot decide their unique interaction weights and thus cannot generalize to noisy inputs.

• Citation, clarification, and typos

We highly appreciate your thorough and constructive feedback on the texts, figures, tables, and citations. We plan to incorporate all of them in our revised manuscript.