SEEKING THE SEARCH SPACE FOR SIZE-AWARE VISION TRANSFORMER ARCHITECTURE

We thanks your valuable comments and feedback. Your raised concerns are addressed in details as follows.

Weakness part

1. Good suggestion! As S3, the proposed method can also be applied to other datasets on larger scales. However, our computational resources can not afford ImageNet-scale experiment evaluation. For more extensive and fairer comparisons, we conduct the experiments in our paper on larger and more complex datasets than Cifar10 and Cifar100, such as Tiny ImageNet and SUN397, which both contain 100,000 images and 200 and 397 classes, respectively. As shown in Table 3 of the paper, for Tiny ImageNet, our approach achieves 75.59% classification accuracy in the search spaces of Shunted Transformer and improves 1.52% in accuracy. Similarly, we achieve 78.19% on the SUN397 dataset and improve 2.09% in accuracy.
2. That's a very good suggestion. In our current work, the step size design is consistent between different datasets, and all datasets achieve excellent performance with this setting. The manual design effort of this consistent step size is low and the scales of the values are referenced from S3. As long as the step size is small enough, with enough exploration iterations, the search space will finally explore to the best one considering the size constraint. However, we will take your suggestion into consideration for the further improvement of our method.
3. Sorry for the insufficient description for this part. Here we provide an example of how we calculate the gradient: Let's say we want to calculate the gradient of the embedding dimension. First we randomly sample a subnet $\alpha$ from the supernet. The depth of $\alpha$ is 12, the embedding dimension is 360, the MLP ratios are all 4 in 12 layers and the number of heads are all 2 in 12 layers. Then we replace the value of the embedding dimension to each of the candidate choices {336, 360, 384}. We then get three architectures sharing the same value on all of the search dimensions except for the embedding dimension. We can acquire the inherited accuracy of them from the supernet, for example, 90, 93 and 97. The gradient of the first two architectures will be $(93 - 90) / (360 - 336) = 0.125$. Similar calculation will be conducted in any two of the three architectures. The above procedure will be conducted 40 times and we finally average all the gradients. Same process is applied to all search dimensions. We also provide the detailed algorithm of the whole process in the Supplementary Material.
4. That's a good question. Though we currently initial search spaces according to exiting architectures, in Section E in the Supplementary material, it can be observed that the overall performance improves with every iteration. Through our constraint optimization algorithm, the search space will continually explore to a better one. As long as the number of the iterations is sufficient, the initial search space will finally stop at the best space align with the user specified size constraint.
5. Thank you for the question. We follow the same setting like S3, which also predefined a range of the search space at the beginning, then apply a search space exploration algorithm to improve them. Compared to S3, we are free from the quantization parameter which needs to be carefully tuned and fully rely on the accuracy gradient ascent algorithm to improve our search space. The manual effort of our method is relatively low.
6. We would like to kindly remind you that the comment seems to be incomplete, and we first respond to the part we currently understand of the problem. As you mentioned, we reference the state-of-the-art architectures known for their strong performance to design our initial search space. Instead of manually adjust these architectures, we introduce a constraint optimization algorithm to automate the process. Also, our approach has the plug-and-play characteristic to any initial search spaces including those that are not delicately designed. We believe after sufficient iterations of exploration, we can finally find the best space aligned with the user-specified size constraint, thereby automating and streamlining the overall design process.
7. As we mentioned in Section 4 in our main paper, the ultimate goal of our research is to find the model with the best performance given predefined model size. To achieve this, we first acquire the desired model size range from users, and we expect them to give us the maximum model size they can afford considering their resources in the real application scenario. Thus, constraining the model size range in the search space exploration phase can largely reduce the extra effort of training the architectures whose size are beyond the user expected. It is a more efficient way compared to first train a large supernet which includes lots of unqualified architectures and lately discard them until the evolutionary search stage.
8. Good question! We will release our code upon paper acceptance.

We thanks your valuable comments and feedback. Your raised concerns are addressed in details as follows.

Weakness part

1. Thanks for the question. To our best knowledge, the most prominent work related to the search space exploration of the transformer architecture search is S3. This is the reason why we have conducted extensive experiments and provided analysis and comparison between S3 and our method. We have proposed a constrained optimization framework combining the accuracy gradient ascent with discrete neighboring search space evolution while adhering to strict model size constraints. Our design avoids extra training and sampling efforts on models that do not align with our size expectations and also performs well with adapters for foundation models, which are new features that S3 does not have.

2. Good suggestion! As S3, the proposed method can also be applied to other datasets on larger scales. However, our computational resources can not afford ImageNet-scale experiment evaluation. For more extensive and fairer comparisons, we conduct the experiments in our paper on larger and more complex datasets than Cifar10 and Cifar100 (both datasets contain images with a resolution of only 32 x 32), such as Tiny ImageNet and SUN397, which both contain 100,000 images and 200 and 397 classes, respectively. As shown in Table 3 of the paper, for Tiny ImageNet, our approach achieves 75.59% classification accuracy in the search spaces of Shunted Transformer and improves 1.52% in accuracy. Similarly, we achieve 78.19% on the SUN397 dataset and improve 2.09% in accuracy.

3. Thank you for the question. Our step size design is consistent between different datasets, and all datasets achieve excellent performance with this setting. The manual design effort of a consistent step size is low and the scales of the values are comparable to those of S3. Rather than being a parameter which need an intensive tuning as the parameter in S3, our method are free from the parameters like that and achieve a better performance at the same time.

We thanks your valuable comments and feedback. Your raised concerns are addressed in details as follows.

Weakness part

1. Thanks for the question. For ViT architecture, the search space exploration takes 12 hours, supernet training takes 3 hours, the evolutionary search takes 0.5 hours, and retraining takes 2.5 hours using a 10-core Intel Xeon CPU with 3GHz and 4 Nvidia Titan V GPUs with 32GB memory on Cifar10 dataset. For the Cifar100 dataset, the search space exploration took 13.5 hours, supernet training took 3.5 hours, evolutionary search took 1 hour, and retraining took 3 hours under the same GPU settings. The search time is similar when we implement the S3 method in the same environment. Though having a similar time cost, we can always find a better architecture than S3 that satisfies the predefined constraint, whereas the quantization hyperparameter in S3 needs to be tuned carefully and requires multiple attempts to make sure to find an architecture aligns with the predefined model size. This proves the efficiency of our method.

2. That’s a very good observation. We would like to kindly remind you that we have provided the inherited accuracy of 300 sampled architectures during each of the search space exploration iteration in Section E in the Supplementary Material. Fig. 5 and Fig. 6 show the results of Swin Transformer and Shunted Transformer on Tiny ImageNet throughout the search space evolution process. It can be seen that the performance of the highest and lowest architectures has improved, leading to an overall upward trend in performance. This once again demonstrates the effectiveness of our method.

3. Good suggestion! As S3, the proposed method can also be applied to other datasets on larger scales. However, our computational resources can not afford ImageNet-scale experiment evaluation. For more extensive and fairer comparisons, we conduct the experiments in our paper on larger and more complex datasets than Cifar10 and Cifar100 (both datasets contain images with a resolution of only 32 x 32), such as Tiny ImageNet and SUN397, which both contain 100,000 images and 200 and 397 classes, respectively. As shown in Table 3 of the paper, for Tiny ImageNet, our approach achieves 75.59% classification accuracy in the search spaces of Shunted Transformer and improves 1.52% in accuracy. Similarly, we achieve 78.19% on the SUN397 dataset and improve 2.09% in accuracy.

Question part

Thank you for the question. The answer is yes, we follow the same setting of S3, which trains a once-for-all supernet to obtain a large amount of well-trained subnets.